Conformal Prediction-Enabled Sequence-level Policy Optimization for Reasoning Language Models
Reinforcement learning improves the reasoning capability of large language models through verifiable rewards for mathematical and coding tasks.
ProofPaper ↗
Key points
- In token-level methods such as GRPO, clipping acts on individual token ratios while the reward and advantage apply to a complete response.
- Recent sequence-level methods align these units but still use fixed clipping bands, whose tail probabilities change as the score distribution evolves.
- We propose Conformal Prediction-Enabled Sequence-level Policy Optimization (CSPO), which adaptively calibrates sequence-level clipping bounds using recent training statistics.
- Experiments with the Qwen3-1.7B series show improvements over GSPO on difficult mathematical reasoning benchmarks and ALFWorld, with clipping endpoints that adapt during training.
Sources (1)
- [1]Conformal Prediction-Enabled Sequence-level Policy Optimization for Reasoning Language ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 07:55 AM
Reinforcement learning improves the reasoning capability of large language models through verifiable rewards for mathematical and coding tasks.
In token-level methods such as GRPO, clipping acts on individual token ratios while the reward and advantage apply to a complete response.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 17, 2026ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning
- Oct 16, 2026Cross-Centaur: Modeling Individual Decision-Making Across Psychological Tasks with Language Models
- Oct 16, 2026Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
- Oct 16, 2026Language Models Learn to Improve Language Models
- Oct 15, 2026AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents
- Oct 15, 2026Towards Continual Learning: Mitigating Forgetting With Subspace Tuning