Same Answer, Different Reasons: Trace-Level Divergence in Multilingual Large Language Models

Key points

• Multilingual large language models are increasingly deployed in settings where visible reasoning traces support verification, education, and safety auditing, yet standard evaluations typically measure only final-answer agreement or holistic response similarity.1
• Across three reasoning benchmarks, seven languages, and six frontier LLMs, the natural baseline---embedding each whole response and comparing the two by cosine---systematically overstates agreement: $52.1%$--$67.3%$ of examples in the top quartile of holistic similarity are not in the top quartile of structural similarity.1
• A same-language control shows that the level of divergence is language-specific: two English traces for paraphrases of the same question agree more closely than an English trace and its counterparts in other languages.1
• Our results show that answer-level and holistic evaluations can substantially overestimate the consistency of multilingual explanatory artifacts, motivating trace-level evaluation for reliable multilingual deployment.1

Sources (1)

• [1]Same Answer, Different Reasons: Trace-Level Divergence in Multilingual Large Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 02:35 AM

  We introduce a trace-level divergence framework that compares visible reasoning traces as structured sequences, decomposing agreement into structural similarity, order consistency, and holistic similarity, with pair-specific calibration to enable comparison across languages.

  Multilingual large language models are increasingly deployed in settings where visible reasoning traces support verification, education, and safety auditing, yet standard evaluations typically measure only final-answer agreement or holistic response similarity.

Extractive summary: sentences quoted from the sources.