Local proxies for memory-efficient training

Key points

• We study how to achieve accurate and memory efficient local learning in transformers.1
• For each transformer block, we train a local proxy network to predict the global network output via distillation.1
• Evaluating the performance of proxy propagation on CIFAR-100 in vision transformers, we find that it achieves a better accuracy/memory trade-off compared to other local algorithms.1
• Finally, we show that the method is further relevant for fine-tuning large language models on a small memory budget.1

Sources (1)

• [1]Local proxies for memory-efficient training
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 08:41 AM

  Local learning algorithms can reshape resource requirements of artificial intelligence.

  We study how to achieve accurate and memory efficient local learning in transformers.

Extractive summary: sentences quoted from the sources.