Do LLMs differentiate linguistic traits internally?: A layer-wise analysis

Key points

• Large language models often produce judgments that resemble human responses, but it remains unclear whether they internally capture linguistic nuances such as conventional versus idiomatic meaning.1
• The analysis combined layer-wise probing with geometric distance measures, using the top embedding dimensions ranked by principal component loadings for each layer.1
• The results show that even with equal capacity, models organize linguistic information in fundamentally different ways, suggesting the need to analyze internal representational geometry, examining not just behavioral outputs but semantic space alignment when evaluating how closely human and model language interpretation correspond.1

Sources (1)

• [1]Do LLMs differentiate linguistic traits internally?: A layer-wise analysis
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 10:28 PM

  A layer-wise representational analysis was conducted on two equal-sized pre-trained models, Llama-3.2-1B and Gemma-3-1B-PT, to examine whether and where these models differentiate linguistic traits from input.

  Large language models often produce judgments that resemble human responses, but it remains unclear whether they internally capture linguistic nuances such as conventional versus idiomatic meaning.

Extractive summary: sentences quoted from the sources.