Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR
Reinforcement learning with verifiable rewards (RLVR) typically uses small rollout groups per prompt, where repeated sampling from the same policy can yield highly similar trajectories, reducing within-group reward variation and weakening group-relative advantage signals.
ProofPaper ↗
Key points
- In this paper, we propose EDEN (Embedding-space Directional Exploration via anchored Noise), which intervenes at the rollout source through anchored input perturbations.
- Selected perturbed rollouts contribute to GRPO updates with downweighted advantages, and the model architecture, discrete decoding, and test-time inference remain unchanged.
- Analyses show that more prompt groups provide non-zero group-relative learning signals and that noise and anchoring play complementary roles.
- Intervening at the rollout source is thus an effective lever for exploration in RLVR.
Sources (2)
- [1]Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVRarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 03:13 AM
Reinforcement learning with verifiable rewards (RLVR) typically uses small rollout groups per prompt, where repeated sampling from the same policy can yield highly similar trajectories, reducing within-group reward variation and weakening group-relative advantage signals.
In this paper, we propose EDEN (Embedding-space Directional Exploration via anchored Noise), which intervenes at the rollout source through anchored input perturbations.
- [2]Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVRHugging Face Daily Papers · Oct 12, 12:00 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 11, 2026Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards
- Oct 11, 2026RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
- Oct 10, 2026KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization
- Oct 9, 2026A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning
- Oct 9, 2026EmoGavel: Grounding-Aware Reinforcement Learning with Verifiable Rewards for Multimodal Emotion Preference Learning
- Oct 9, 2026AdaC-GRPO: Towards Mastering Long-Horizon Visual Sudoku Puzzles via Adaptive Curriculum Learning