LongZero: Enhancing Long-Context Reasoning in LLMs from Zero Data
Therefore, we propose $LongZero$, a method that lets an LLM improve its own long-context reasoning without any external data.
ProofPaper ↗
Key points
- Reinforcement learning (RL) is an effective way to strengthen the long-context reasoning of large language models (LLMs), but it needs training data that are long, diverse, challenging, and verifiable.
- LongZero combines these self-written multi-hop reasoning tasks with self-written multi-round retrieval tasks and trains the model with GRPO.
- We evaluate LongZero on two LLMs across five long-context reasoning benchmarks and compare it with RL on data synthesized from external corpora.
- Experimental results show that LongZero can generalize to real tasks, and outperforms both baselines, suggesting a viable and promising path to improving long-context reasoning of LLMs.
Sources (1)
- [1]LongZero: Enhancing Long-Context Reasoning in LLMs from Zero DataarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 05:59 PM
Therefore, we propose $LongZero$, a method that lets an LLM improve its own long-context reasoning without any external data.
Reinforcement learning (RL) is an effective way to strengthen the long-context reasoning of large language models (LLMs), but it needs training data that are long, diverse, challenging, and verifiable.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 21, 2026EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping
- Oct 21, 2026Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs
- Oct 20, 2026When Do LLMs Help with Feature Selection? A Systematic Empirical Study
- Oct 20, 2026RANGER: Retrieval Agent for Navigating Graphs with Semantic Evaluation Rewards
- Oct 20, 2026Hallucinations of Large Language Models in Cross-Language Code Translation: Phenomena and Mitigation
- Oct 20, 2026Do Diffusion LLMs Scale? Testing Mercury 3 + Decide