One-Layer Transformers Provably Learn In-Context K-Nearest Neighbor Prediction with Chain-of-Thought

Key points

• In this work, we study this question in a stylized yet fully analyzable setting: in-context $K$-nearest-neighbor ($K$-NN) prediction with a one-layer transformer.1
• Earlier work has demonstrated that one-layer transformers can be trained to perform $1$-NN without CoT (Li et al., 2024a) by having the softmax attention attend to the nearest neighbor in context.1
• We then give an explicit construction of a one-layer transformer, and show that it can solve $K$-NN via CoT reasoning.1
• In addition, we show that the resulting trained model applies to a broader test class than that assumed in the training analysis.1

Sources (1)

• [1]One-Layer Transformers Provably Learn In-Context K-Nearest Neighbor Prediction with Chain-of-Thought
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 01:50 PM

  Chain-of-Thought (CoT) enables transformers to solve complex reasoning tasks by generating intermediate reasoning steps.

  In this work, we study this question in a stylized yet fully analyzable setting: in-context $K$-nearest-neighbor ($K$-NN) prediction with a one-layer transformer.

Extractive summary: sentences quoted from the sources.