In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model

Key points

• Sequence models can generalise in fundamentally distinct ways: extrapolating structures seen in training, or using their prompt to adapt to novel tasks.1
• Recent work has characterised many different strategies theoretically, and empirically described the transition between them as the quality and quantity of pretraining data vary.1
• The dataset comes with two distinct solutions: a specialised predictor, that stores the finite topic mixture, and an in-context one, optimal for topics never seen during training.1
• Our theory predicts their emergence at distinct sample complexities: the in-context predictor is available early, while the specialised one emerges only after the training tokens outnumber the parameters of the data-generating process.1

Sources (1)

• [1]In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 01:54 PM

  Here we provide such a theory in a solvable topic model---documents drawn from a finite mixture of latent topics, mirroring the heterogeneity of modern pretraining data.

  Sequence models can generalise in fundamentally distinct ways: extrapolating structures seen in training, or using their prompt to adapt to novel tasks.

Extractive summary: sentences quoted from the sources.