RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning

Key points

• A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO.1
• RELAY uses advantages and rewards to select positive-advantage and low-scoring zero-advantage rollouts, and monitors the distillation loss to decide when to retire the teacher.1
• Stage 1 combines selective distillation from a frozen teacher with GRPO; a plateau detector then removes the teacher, and Stage 2 continues with GRPO alone.1
• RELAY scores highest in-domain among the compared methods in all four settings, ahead of GRPO on RaR-Medicine by 4.62 points with Qwen3-1.7B and 6.02 with Qwen3-4B.1

Sources (1)

• [1]RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 02:45 AM

  Rubric-based reinforcement learning grades a response against many written criteria but collapses them into one reward, which discards the detail the rubric carries and gives no GRPO reward gradient when a group's responses tie.

  A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO.

Extractive summary: sentences quoted from the sources.