ResearchResearch paperLarge Language Models · Efficiency & Inference1 source · Oct 13, 2026

Prefilling-dLLM v2: Learned Chunk Routing for Long-Context Diffusion Language Models

We propose Prefilling-dLLM v2, which continues training a diffusion language model so that it routes with its native mask token.

Diffusion language models apply bidirectional attention over the whole context at every denoising step, so a longer context makes each step substantially more expensive.
Sparsifying only at inference leaves the pretrained weights unchanged, so the model is never trained under the sparse pattern it runs with.
One mask is appended at the end of each chunk, and a learned role offset marks it as a routing slot.
Starting from Dream-v0-Base-7B extended to 16K with YaRN, we compare Prefilling-dLLM v2 with bidirectional NSA, DSA, and SWA under the same layout, key budget, and dense initialization, using the same Fast-dLLM decoder.

[1]Prefilling-dLLM v2: Learned Chunk Routing for Long-Context Diffusion Language Models
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 07:29 AM
We propose Prefilling-dLLM v2, which continues training a diffusion language model so that it routes with its native mask token.
Diffusion language models apply bidirectional attention over the whole context at every denoising step, so a longer context makes each step substantially more expensive.

Extractive summary: sentences quoted from the sources.

Before this