SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models

Key points

• Current evaluation of LLMs on benchmarks typically aggregates performance over individual examples into a single scalar score.1
• The framework integrates advances from two areas: (1) it leverages the meta-cognitive capabilities of LLMs to automatically discover interpretable skills from questions, and (2) it applies cognitive diagnosis models (CDMs) from psychometrics to estimate per-skill abilities of LLMs. We conduct experiments on 3,811 LLMs evaluated on 9,523 items across five benchmarks spanning diverse domains (MATH, BBH, GPQA, MuSR, IFEval).1
• We also demonstrate the utility of SkillEval in two downstream tasks: (1) LLM routing, which dynamically assigns each query to a capable but efficient model, reducing inference cost by over 60% while achieving improved performance, and (2) Computerized Adaptive Testing (CAT), where an unseen model can be efficiently evaluated using a trained CDM.1
• Furthermore, we show that these skill profiles provide useful insights, for example, how benchmarks share skills and how instruction-tuned models differ from their base models.1

Sources (1)

• [1]SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 06:06 PM

  To address this limitation and better exploit benchmark data, we propose SkillEval, a new evaluation framework that performs cross-model, skill-level analysis.

  Current evaluation of LLMs on benchmarks typically aggregates performance over individual examples into a single scalar score.

Extractive summary: sentences quoted from the sources.