The Dichotomy Between Pattern Recognition and Step-by-Step Reasoning

Key points

• A large language model (LLM) learns to reason step-by-step when data is structured such that the next token depends on a small amount of preceding context.1
• Inference in LLMs resembles pattern recognition when the next token depends on a large amount of preceding context.1
• If the next token depends on only the $c$ most recent tokens, reasoning traces are paths on a De Bruijn graph whose nodes are $c$-length contexts and edges are next-token transitions between contexts.1
• We show that real-world data has De Bruijn structure: Qwen3-14B and Qwen3-32B retain over 75% of their accuracy on GSM8K, MATH-500 and GPQA-Diamond when attention is restricted to a sliding window less than 15% as long as the full reasoning trace.1

Sources (1)

• [1]The Dichotomy Between Pattern Recognition and Step-by-Step Reasoning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 10:36 PM

  We argue that pattern recognition and step-by-step reasoning are two ends of a spectrum.

  A large language model (LLM) learns to reason step-by-step when data is structured such that the next token depends on a small amount of preceding context.

Extractive summary: sentences quoted from the sources.