Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining

Key points

• This raises a practical question: when RL fails, what is the base model missing?1
• We look inside the RL policy's own rollouts and find that failures are localized: whether a rollout fails is strongly predicted by a small set of characteristic n-grams, ranging from reflection phrases such as "let's recheck" to specific operations such as "the inverse of A".1
• Building on this observation, we use the failure $n$-grams to retrieve text from the pretraining corpus and continue pretraining the base model on it before RL.1
• We evaluate this approach on OLMo2 and Qwen3 models trained with RL on (1) GSM8K; (2) a mixture of math, commonsense, and knowledge questions; and (3) general math and science reasoning.1

Sources (1)

• [1]Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:48 AM

  Reinforcement learning with verifiable rewards (RLVR) is widely used to improve reasoning in language models, yet growing evidence suggests that it mostly sharpens what the base model can already do.

  This raises a practical question: when RL fails, what is the base model missing?

Extractive summary: sentences quoted from the sources.