Your Transformer Is Secretly Elastic

Key points

• Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads.1
• Our key contribution is a differentiable relaxation of the functional unit-ranking problem: unlike prior work, which optimizes a single importance weight per MLP block or attention head, we rank every individual MLP neuron and attention head jointly, on a common scale, by minimizing the model's loss under soft pruning across many sparsity targets, all at once.1
• Across 27 vision and text transformer encoders ranging from 22M to 8B parameters, our method yields smooth degradation up to 60% sparsity and outperforms one-shot baselines under deep pruning.1
• At 60% sparsity, a pruned AugReg ViT-B/16 model achieves up to $1.80\times$ higher throughput and $1.26\times$ lower single-query latency on an A100 GPU.1

Sources (2)

• [1]Your Transformer Is Secretly Elastic
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 05:58 PM

  We introduce a retraining-free method that elastifies pre-trained transformers into a single model that adapts on-the-fly across a continuum of computational budgets.

  Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads.

• [2]Your Transformer Is Secretly Elastic
  Hugging Face Daily Papers · Oct 14, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.