Wrong Votes, Right Gradients: Why Majority-Vote Rewards Track Verifiable RL

Key points

• Reinforcement learning with self-generated labels, such as majority vote over a model's own samples, often recovers most of the gain of training with verifiable rewards.1
• This is puzzling from the usual prompt-level view, in which majority vote only returns a model's modal answer and cannot add information about correctness.1
• Because parameters are shared, a prompt's own gradient accounts for less than 1% of the change in that prompt's output at practical batch sizes, so a wrong label matters only to the extent that label errors share a direction across prompts.1
• We decompose the majority-vote update into shrinkage and misdirection relative to the gold update, and show that misdirection vanishes with batch size whenever errors are incoherent across prompts, however often the labels are wrong.1

Sources (1)

• [1]Wrong Votes, Right Gradients: Why Majority-Vote Rewards Track Verifiable RL
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 08:25 PM

  Measuring per-response gradients on five models, we find that majority-vote errors are as incoherent as a random-reward null, that the majority-vote update agrees with the gold update as closely as two independent gold updates agree with each other, and that its noise-corrected cosine with the gold update is 1 within error.

  Reinforcement learning with self-generated labels, such as majority vote over a model's own samples, often recovers most of the gain of training with verifiable rewards.

Extractive summary: sentences quoted from the sources.