The Reliance Hypothesis: Chain-of-Thought Faithfulness Rises as Language Resource Level Falls

Key points

• Chain-of-Thought (CoT) faithfulness matters for monitoring, explaining and safely deploying large language models, yet it has been studied mostly in English.1
• Languages differ in level, the amount of text available to learn from, so models know less in lower-resource languages, which may change how much they rely on their reasoning.1
• On parallel multiple-choice questions from Multilingual Massive Multitask Language Understanding (MMMLU) in eight languages, faithfulness rises as resource level falls on knowledge-intensive tasks but not on reasoning-intensive ones, a gap that points the same way across four models and several faithfulness measures.1
• Model confidence partly explains the effect of resource level, but not the knowledge-reasoning gap.1

Sources (1)

• [1]The Reliance Hypothesis: Chain-of-Thought Faithfulness Rises as Language Resource Level Falls
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 02:33 AM

  We propose, and find support for, the Reliance Hypothesis: models rely more on their reasoning when they cannot recall the answer, as in lower-resource languages on knowledge-intensive tasks.

  Chain-of-Thought (CoT) faithfulness matters for monitoring, explaining and safely deploying large language models, yet it has been studied mostly in English.

Extractive summary: sentences quoted from the sources.