Can Capability Enhancement be Localized?
In this work, we investigate a complementary aspect, called Capability Enhancement Localization (CEL): given a model and a task, does there exist a small set of heads whose removal improves the task performance?
ProofPaper ↗
Key points
- Transformer-based language models have shown strong capabilities across many tasks and domains.
- Using a simple two-stage routine that screens candidate heads and evaluates their joint ablations, we identify sparse head combinations whose removal improves task-specific performance without further fine-tuning.
- To this end, we construct a simplified multi-head softmax attention model and two tasks with conflicting features, and show that gradient descent from small random initialization provably concentrates the conflicting features into particular heads.
- We further show that ablating such heads improves performance on one task while harming the other, even though training converges to a global optimum of the joint objective.
Sources (1)
- [1]Can Capability Enhancement be Localized?arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 05:41 AM
In this work, we investigate a complementary aspect, called Capability Enhancement Localization (CEL): given a model and a task, does there exist a small set of heads whose removal improves the task performance?
Transformer-based language models have shown strong capabilities across many tasks and domains.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
- Oct 15, 2026Welcome FLUX.3 Image Klein in Diffusers
- Oct 14, 2026Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learning
- Oct 14, 2026mistralai/Timestral-209M
- Oct 14, 2026Eye, Robot: Learning Expressive Gaze Dynamics via a VLM-Guided Transformer-VAE
- Oct 14, 2026DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections
