What Do Sparse Autoencoder Feature Discovery Scores Identify? From Activations to Interventions

Key points

• For a broad class of response functions, we show that observational feature-discovery evidence generally does not identify this attribution, and derive an exact gradient-based decomposition of the gap between linear-probe predictions and intervention attribution.1
• Across four model-SAE pairs, median probe-to-attribution magnitude ratios range from tens to hundreds, while attributions closely track small-step steering.1
• Attribution-selected sets also produce larger absolute mean ablation effects on held-out prompts than sets selected by correlation, rank tests or probe coefficients.1
• Together, these results establish intervention attribution as a principled, gradient-estimable target for selecting SAE features for intervention.1

Sources (1)

• [1]What Do Sparse Autoencoder Feature Discovery Scores Identify? From Activations to Interventions
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 07:07 PM

  Sparse autoencoder (SAE) features are typically discovered from how their activations co-vary with behaviour, yet they are often used through interventions such as steering or ablation.

  For a broad class of response functions, we show that observational feature-discovery evidence generally does not identify this attribution, and derive an exact gradient-based decomposition of the gap between linear-probe predictions and intervention attribution.

Extractive summary: sentences quoted from the sources.