When Thoughts Become Memory: Unifying In-Context Learning and Fine-Tuning

Key points

• That value, the code length saved on held-out queries, equals in expectation the bits stored about the query, plus a prior repair, minus a decoding loss, for any model, architecture, dataset and task.1
• The test picks TabPFN over XGBoost and a trained surrogate over an in-context learner on airfoil pressure, calls the better of GPT-2's cache and fine-tune in 61,948 of 64,000 query streams over 32 comparisons, and errs in at most 3.3% of null streams.1
• The law also diagnoses when learning in context helps and when it does not.1
• Ranked by this label-free valuation, a kriging learner's three best support cars beat four drawn at random, and on a yes-or-no claim the valuation recovers a published evidence adjudicator, with AUROC 0.91.1

Sources (1)

• [1]When Thoughts Become Memory: Unifying In-Context Learning and Fine-Tuning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 08:44 AM

  We are the first to give a certified rule for whether unseen data belong in a pretrained model's context or its weights, built on an exact law for their information value.

  That value, the code length saved on held-out queries, equals in expectation the bits stored about the query, plus a prior repair, minus a decoding loss, for any model, architecture, dataset and task.

Extractive summary: sentences quoted from the sources.