Is it RL or Data? A Controlled Comparison of GRPO and Rejection-Sampling Fine-Tuning

Key points

• They did not test whether this benefit is specific to GRPO, or whether another post-training method given the same data would obtain it too.1
• We test this on their procedural counting task with Qwen3-4B-Base and LoRA, comparing GRPO to rejection-sampling fine-tuning (RFT) on the same prompts, verifier and token cap.1
• Six primary arms cross the two methods with three prompt sets (Easy, Mixed, and a Curated subset of prompts the base model sometimes solves, which makes GRPO's implicit filter explicit).1
• GRPO outperforms RFT at every prompt set: by 10.9 points on held-out test problems with curated prompts, and, in post hoc comparisons with matched rollout budgets, by 8.4 and 9.2 points on the easy and mixed prompts.1

Sources (1)

• [1]Is it RL or Data? A Controlled Comparison of GRPO and Rejection-Sampling Fine-Tuning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:26 AM

  Bauer et al. (2026) showed that in low-data reinforcement learning with verifiable rewards, training GRPO on mixed-difficulty prompts gives up to 5x the sample efficiency of easy-only prompts.

  They did not test whether this benefit is specific to GRPO, or whether another post-training method given the same data would obtain it too.

Extractive summary: sentences quoted from the sources.