ARBOR: Locating Reasoning Failures in Long Chains of Thought through Reasoning State Tracking

Key points

• Large reasoning models solve complex problems through long chains of thought, in which prior work has documented failure modes such as overthinking, underthinking, and unsupported deductions.1
• Existing methods detect these failures from local signals: textual similarity to nearby steps, switch tokens, or an LLM's judgment of a step's correctness.1
• Using ARBOR, we refine the three failure modes into distinct subtypes that harm reasoning through different mechanisms and at different costs.1
• We discuss how ARBOR integrates with existing mitigations at inference time and in training data.1

Sources (1)

• [1]ARBOR: Locating Reasoning Failures in Long Chains of Thought through Reasoning State Tracking
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 10:39 PM

  Therefore, we propose ARBOR, a reasoning-failure attribution framework that represents each trajectory as a tree of reasoning states, built as each step creates, verifies, prunes, or revisits a state, and attributes each failure to a relation between a step and the state preceding it.

  Large reasoning models solve complex problems through long chains of thought, in which prior work has documented failure modes such as overthinking, underthinking, and unsupported deductions.

Extractive summary: sentences quoted from the sources.