Looking Inside LLMs: Small-World Connectivity as a Signature of Reasoning Performance

Key points

• Understanding large language model (LLM) reasoning requires looking beyond behavioral performance to examine how reasoning ability is reflected in internal organization.1
• We construct functional graphs from attention-head activation similarities and find that a higher small-world index (SWI), capturing local clustering and short global paths, consistently correlates with better fluid reasoning performance across models and training checkpoints.1
• Since local clustering is central to small-world organization, we further examine how heads important for model performance connect within and across communities.1
• We validate this hypothesis through pruning, introducing Small-World Allocation (SWA), a hierarchical sparsity allocation method guided by these scores.1

Sources (1)

• [1]Looking Inside LLMs: Small-World Connectivity as a Signature of Reasoning Performance
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 04:53 PM

  Inspired by neuroscience findings linking higher intelligence to stronger small-world organization in functional brain networks, we investigate small-world connectivity as a structural signature of LLM reasoning.

  Understanding large language model (LLM) reasoning requires looking beyond behavioral performance to examine how reasoning ability is reflected in internal organization.

Extractive summary: sentences quoted from the sources.