DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections

Key points

• Beyond the residual connections in a standard Transformer, DAGFormer adds direct connections from earlier layers to later ones, and the predictor controls the strength of every connection based on input tokens.1
• Unlike existing cross-layer routing architectures, which set each layer's connections separately, DAGFormer plans the routing of the whole model at once.1
• On matched OLMo-2 pairs from 75M to 1B parameters, DAGFormer lowers WikiText-2 perplexity by 17% to 25% over the vanilla Transformer, and a 1B DAGFormer trained on 5B tokens outperforms the vanilla model trained on 10B.1
• The gain carries over to 13 commonsense, science, reading and math evaluation tasks (2.0 to 2.5 points higher mean accuracy at 300M and 1B) and to instruction finetuning (63% to 70% AlpacaEval win rate against the vanilla Transformer, leading every routing baseline), and under structured pruning DAGFormer degrades the least in five of six settings.1

Sources (2)

• [1]DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 05:50 AM

  We propose DAGFormer, a Transformer language model whose cross-layer connections are planned by a jointly trained topology predictor.

  Beyond the residual connections in a standard Transformer, DAGFormer adds direct connections from earlier layers to later ones, and the predictor controls the strength of every connection based on input tokens.

• [2]DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections
  Hugging Face Daily Papers · Oct 14, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.