DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia

Key points

• While language models have demonstrated remarkable capabilities, their performance remains disproportionately skewed towards high-resource languages.1
• Extensive evaluation of frontier proprietary and open-weights models reveals that while frontier models like Gemini-3.7-Flash maintain robustness across resource tiers, smaller models face severe performance degradation on low-resource languages.1
• Crucially, models perform better in reasoning and summarization when pivoted through English rather than target languages, exposing a critical dependency on English representations, especially for low-resource languages.1
• Beyond these observations, we provide a cross-lingual error analysis and a cross-script tokenizer fertility analysis.1

Sources (1)

• [1]DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 10:43 AM

  To address this, we introduce DeepSEA, a comprehensive benchmark constructed from established datasets through professional human translation into 12 Southeast Asian languages, spanning both high-resource (e.g., Indonesian, Thai, Vietnamese) and low-resource (e.g., Burmese, Khmer, Lao) tiers across five distinct tasks: mathematical reasoning, scientific reasoning, question answering, abstractive s

  While language models have demonstrated remarkable capabilities, their performance remains disproportionately skewed towards high-resource languages.

Extractive summary: sentences quoted from the sources.