Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoning
Exploration bounds learning efficiency in reinforcement learning with verifiable rewards (RLVR).
ProofPaper ↗
Key points
- Current methods explore mostly through independent sampling, without fully using the information gathered along the way, such as known wrong answers and environment feedback.
- We propose LEAF, which injects a dynamic, problem-specific memory into the context before every rollout.
- We decouple policy updates from rollouts and update the policy through self-distillation.
- On four benchmarks and two open models, LEAF accelerates exploration by about 4x compared with a random sampling baseline.
Sources (1)
- [1]Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM ReasoningarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:13 AM
Exploration bounds learning efficiency in reinforcement learning with verifiable rewards (RLVR).
Current methods explore mostly through independent sampling, without fully using the information gathered along the way, such as known wrong answers and environment feedback.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter
- Oct 12, 2026CloneOT: predicting clone-specific perturbation responses with lineage-supervised neural inverse optimal transport
- Oct 12, 2026Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards
- Oct 12, 2026TPPO: Tracking-Value PPO For Language Model Reasoning
- Oct 11, 2026RINO: Renormalization Group Distillation with No Labels
- Oct 11, 2026Apertus 1.5 Release Notes