Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation

Key points

• Diffusion language models (DLMs) offer parallel sampling for faster generation than autoregressive models, an increasingly important advantage as inference becomes a dominant computational workload.1
• At matched parameter scales, NCM achieves better generative perplexity than the strongest DLM baseline on LM1B while running 21times faster.1
• On OpenWebText, NCM is competitive with the strongest DLM while running 10times faster; on XSum, it matches the strongest DLM's ROUGE-L while being 31times faster.1
• On the reasoning task of Sudoku, NCM outperforms autoregressive and diffusion baselines across all difficulty levels, using fewer function evaluations.1

Sources (1)

• [1]Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation
  Hugging Face Daily Papers · Oct 9, 12:00 AM

  We introduce Next-Concept Modeling (NCM), a two-stage, coarse-to-fine framework: a multiscale residual VQ-VAE encodes text into a hierarchy of discrete codes, and a transformer generates one level at a time, predicting all codes within a level in parallel.

  Diffusion language models (DLMs) offer parallel sampling for faster generation than autoregressive models, an increasingly important advantage as inference becomes a dominant computational workload.

Extractive summary: sentences quoted from the sources.