Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation

Key points

• Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in English.1
• Translating existing benchmarks offers a scalable approach to multilingual evaluation, but preserving task-specific meaning across languages presents significant challenges, particularly for harmful requests, mathematical reasoning, and conversational interactions.1
• Using this framework, we develop a source-aligned evaluation collection spanning seven safety, conversational, and mathematical reasoning resources across 11 languages.1
• Evaluations of open-source LLMs reveal distinct language-dependent vulnerabilities, including inconsistent refusal behaviour and substantial reasoning degradation in lower-resource languages.1

Sources (1)

• [1]Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 01:58 PM

  We introduce a reasoning-guided translation framework for constructing multilingual evaluations designed to preserve the structure and intended meaning of diverse benchmark tasks.

  Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in English.

Extractive summary: sentences quoted from the sources.