OrderGPT: Generation Order as a Learnable Modeling Choice

Key points

• For structured discrete domains without a natural traversal, both extremes may be suboptimal, yet the intermediate choice -- a single well-adapted order -- remains largely unexplored.1
• We introduce OrderGPT, which builds on Any-Order GPT to learn a task-level generation order without canonical-order supervision.1
• Our key observation is that predicting variables from randomly sampled reveal prefixes produces recurring attention patterns in the backbone that provide a model-dependent signal of predictive dependencies.1
• A lightweight OrderMLP integrated into the GPT backbone learns to predict these priorities from attention, incorporating order learning into the architecture within a single training run.1

Sources (1)

• [1]OrderGPT: Generation Order as a Learnable Modeling Choice
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 07:41 AM

  Generative modeling of discrete data has two popular paradigms: autoregressive models such as GPT, which factorize a sequence in a left-to-right (L2R) order, and masked diffusion models, which are essentially any-order autoregressive modeling.

  For structured discrete domains without a natural traversal, both extremes may be suboptimal, yet the intermediate choice -- a single well-adapted order -- remains largely unexplored.

Extractive summary: sentences quoted from the sources.