Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures

Key points

• Adversarial training is one of the most reliable defenses against adversarial attacks, but its high computational cost must generally be paid anew for each task.1
• Robust foundation models offer a promising alternative: adversarially pretrain a model once and then transfer its robustness to downstream tasks through lightweight adaptation.1
• However, a fundamental question remains open: can robustness acquired during pretraining transfer to unseen tasks without further adversarial training?1
• A single model adversarially pretrained at scale can achieve optimal robustness on new tasks without additional task-specific training.1

Sources (1)

• [1]Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 04:51 AM

  Specifically, we show that, for a family of Gaussian-mixture classification tasks, a sufficiently deep linear transformer adversarially trained across tasks can asymptotically attain the robust Bayes error on previously unseen tasks through in-context learning from clean demonstrations.

  Adversarial training is one of the most reliable defenses against adversarial attacks, but its high computational cost must generally be paid anew for each task.

Extractive summary: sentences quoted from the sources.