Parallel Language Models via Variational Decoding
Autoregressive language models emit one token per forward pass.
ProofPaper ↗
Key points
- In this paper, we introduce Parallel Language Models (PLMs), which generate mutually consistent tokens in one forward pass by routing dependencies through latent auxiliary variables.
- PLMs are trained using a variational bound that reduces to an autoregressive loss plus a parallel decoding loss.
- When post-trained, our 1.3B model matches or exceeds an identically trained autoregressive model on math, coding, and instruction-following benchmarks, speeding up inference $2.3-5.7\times$.
- Together, we show that parallel generation can be trained from a single training objective at autoregressive quality.
Sources (1)
- [1]Parallel Language Models via Variational DecodingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 04:34 PM
Autoregressive language models emit one token per forward pass.
In this paper, we introduce Parallel Language Models (PLMs), which generate mutually consistent tokens in one forward pass by routing dependencies through latent auxiliary variables.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026TorchNeuron: running HF models on AWS Trainium is easier than ever
- Oct 19, 2026From Short Clips to Real-Time Streams: Identity-Aware Supervision and Context-Efficient Generation for Video Face Swapping
- Oct 19, 2026HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL
- Oct 19, 2026InfiniDream: Open-Ended Photorealistic 3D Worlds Through Procedural Generation
- Oct 18, 2026VLA-Rolling: Fewer Denoising Steps, Better Control via Adaptive Action Reuse
- Oct 16, 2026[Research] 2,788 tokens/s on a single RTX 5090 with a diffusion LLM — Introducing ORL (fully open source)