The Thinking Leakage in Reinforcement Learning for LLMs

Key points

• A popular recipe trains such models with reinforcement learning in NO-THINK mode and reports improved reasoning performance.1
• We provide evidence that much of this gain can be explained by thinking leakage: the gain comes largely from partially reactivating the reasoning abilities of the original model's THINK mode instead of learning enhanced reasoning behavior.1
• We train Qwen3-1.7B, 4B, and 8B with GRPO in NO-THINK mode.1
• We further identify a possible mechanism in the GRPO training signal: responses favored by the advantage-weighted update are more THINK-like, and a gradient step computed from this signal moves the model toward its THINK state.1

Sources (2)

• [1]The Thinking Leakage in Reinforcement Learning for LLMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 07:43 PM

  Hybrid reasoning models such as the Qwen3 series switch between a THINK mode and a NO-THINK mode through a few chat-template tokens.

  A popular recipe trains such models with reinforcement learning in NO-THINK mode and reports improved reasoning performance.

• [2]The Thinking Leakage in Reinforcement Learning for LLMs
  Hugging Face Daily Papers · Oct 9, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.