Large Flow Language Models

Key points

• Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet effective techniques for building them at the billion-parameter scale remain less established.1
• Lafa is a blockwise flow model with causal attention on the prefix and bidirectional, asynchronous flow within each block.1
• We train Lafa at 0.5B, 7B, and 8B scales with fewer than 30B tokens, more efficient than previous diffusion models, and evaluate it on instruction following, reasoning, and coding, where the learned flow matches or surpasses the quality of AR and can be distilled into flow maps for one- and few-step generation.1
• Notably, flow training preserves or even improves pretrained AR capabilities via a multi-token prediction effect, which we leverage in a self-speculative decoding scheme that verifies flow-based drafts autoregressively within a single model.1

Sources (2)

• [1]Large Flow Language Models
  Hugging Face Daily Papers · Oct 20, 12:00 AM

  We present Lafa (Large flow language model), a fast, high-performing flow language model scaled to 8B parameters in a data-efficient manner.

  Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet effective techniques for building them at the billion-parameter scale remain less established.

• [2]Large Flow Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 05:20 PM · same content

Extractive summary: sentences quoted from the sources.