Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning

Key points

• To understand the attention mechanism in transformer blocks, a growing line of theoretical work interprets ICL through the lens of classical statistical estimators.1
• In this work, we demonstrate that transformers can act as adaptive anisotropic kernel learners.1
• Specifically, we study ICL for low-rank linear and multi-index nonlinear regressions, and conduct numerical simulations to show that the learned attention scores closely resemble the oracle kernels for these function classes.1
• Our theory gives an explicit and constructive implementation of the oracle kernel estimators for these function classes using a wide and shallow transformer architecture, illustrating how the prediction is carried out by the interaction between the query token and the context tokens.1

Sources (1)

• [1]Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 04:01 AM

  Transformers have emerged as a dominant architecture in modern machine learning, with in-context learning (ICL) enabling models to adapt to new tasks by conditioning on prompts, without any parameter updates.

  To understand the attention mechanism in transformer blocks, a growing line of theoretical work interprets ICL through the lens of classical statistical estimators.

Extractive summary: sentences quoted from the sources.