Why don't Graph Transformers attend to the same token?
In large language models (LLMs), attention concentrates heavily on the first token.
ProofPaper ↗
Key points
- This 'attention sink' is believed to protect deep models against over-mixing.
- We find that the attention scoring function is the main cause.
- In Graph Transformers using dot-product attention, removing the sink gives about the same error as removing a random node, while GRIT's most-attended nodes matter to its predictions.
- Attention sinks are therefore a property of dot-product attention rather than of Transformers.
Sources (1)
- [1]Why don't Graph Transformers attend to the same token?arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 12:02 AM
In large language models (LLMs), attention concentrates heavily on the first token.
This 'attention sink' is believed to protect deep models against over-mixing.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios
- Oct 13, 2026Narrowing the Collaboration Gap, Probably
- Oct 13, 2026Plan or Retrieve First? A Scale Study of Tool Discovery Architectures
- Oct 13, 2026Read the Room Before Your Model Generates
- Oct 13, 2026AIRRI: Towards a Human-Readable, Machine-Resistant Reading Interface
- Oct 12, 2026From Parallel to Sequential: Small-Batch Noise Restores Simplicity Bias of Muon