Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering

Key points

• Transformers are deployed as a special model class.1
• Every Transformer block, with attention, LayerNorm and a GELU MLP, compiles exactly into a generic deep neural network built from fixed affine maps, routing and the square, sigmoid and inverse square root, plus GELU or its sigmoid approximation.1
• Without activation-activation products, every transform-wise compilation adds at least $N^2$ scalar nonlinear evaluations per head for dense scores.1
• For top-$k$ attention with exchangeable tail scores, we prove that exact or conditional-mean tail mass gives an unbiased, minimum-mean-square-error estimate of the attention distribution, while tail drop is conditionally biased.1

Sources (1)

• [1]Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 10:16 PM

  Neural processing units (NPU) add dedicated operators for attention and normalization, and quantization relies on Transformer-specific algorithms.

  Transformers are deployed as a special model class.

Extractive summary: sentences quoted from the sources.