How Language Models Organize Task Information

Key points

• We measure readout sparsity with \(k^\), the smallest feature count found by a fixed search protocol that recovers 95% of dense-probe AUROC above chance.1
• In controlled synthetic experiments across 20 models, successfully learned targets admit single-feature readouts, whereas readable controls require tens of features.1
• Reference-model \(k^\) is associated with label requirements across eight students, following an empirical scaling of \(N^\propto(k^)^{0.65}\).1
• These findings distinguish task readability from readout sparsity and establish their complementary value for predicting learning requirements and diagnosing LM changes.1

Sources (1)

• [1]How Language Models Organize Task Information
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 06:43 AM

  Dense linear probes assess task readability from language model (LM) representations, but do not directly reveal how many features suffice for prediction.

  We measure readout sparsity with \(k^*\), the smallest feature count found by a fixed search protocol that recovers 95% of dense-probe AUROC above chance.

Extractive summary: sentences quoted from the sources.