ResearchResearch paperReinforcement Learning · Large Language Models · Applications2 sources · Oct 13, 2026

Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders

When an LLM recommender is trained on such threads with reinforcement learning (RL), which replies should a generated list be rewarded against?

A recommendation request on Reddit draws several replies, each naming different titles.
The state-of-the-art recipe, Rank-GRPO, picks one reply per training example, so the advantage mostly records which reply was drawn (85% of its variance) rather than how good the list is.
We argue that RL should train on the request, not on each answer, and propose ReMaRk: it scores every list once against the union of all replies' titles and samples each request equally.
On seven backbones and three datasets, ReMaRk reaches the validation recall of 2,000 per-reply RL steps in 25 to 53% of the steps and leads at that budget; trained until neither improves, it is never significantly behind Rank-GRPO and in four of seven pairs reaches Rank-GRPO's best with fewer GPU-hours.

[1]Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:57 AM
When an LLM recommender is trained on such threads with reinforcement learning (RL), which replies should a generated list be rewarded against?
A recommendation request on Reddit draws several replies, each naming different titles.
[2]Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders
Hugging Face Daily Papers · Oct 13, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.

Before this