Learning to Reason with Selective Internal Feedback
Reinforcement learning from internal feedback can improve language models without ground-truth answers or external verifiers.
ProofPaper ↗
Key points
- However, confidence and correctness can diverge during training, and continued optimization of average self-certainty can increase confidence while degrading reasoning accuracy.
- We introduce SIF, a reinforcement learning method that combines comparative reward construction with selective policy updates.
- SIF compares ranked token-level self-certainty profiles across multiple windows and peers, assigning positive rewards only when both levels of voting provide sufficient support.
- Experiments on three backbones across 12 mathematical and scientific reasoning benchmarks show that SIF's final checkpoints achieve higher average accuracy than the best-validation checkpoints of Intuitor and Vigor, with a gain of 4.71 percentage points over Intuitor on Qwen2.5-Math-1.5B.
Sources (1)
- [1]Learning to Reason with Selective Internal FeedbackarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 07:08 AM
Reinforcement learning from internal feedback can improve language models without ground-truth answers or external verifiers.
However, confidence and correctness can diverge during training, and continued optimization of average self-certainty can increase confidence while degrading reasoning accuracy.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026PocketTSLM: Deploying an On-Device Soft-Prompt Time-Series Language Model for Privacy-Preserving Smartphone Reasoning
- Oct 15, 2026The Density of Thought: Predicting and Steering Reasoning in Large Language Models
- Oct 15, 2026Learning to Think and Explore in 3D World
- Oct 15, 2026Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models
- Oct 15, 2026Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries
- Oct 15, 2026Mosaic: Multi-Session Agentic Reinforcement Learning