Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

Key points

• Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally.1
• However, existing methods commonly aggregate the verification results with uniform weighting, which assigns minor and overlapping criteria the same weight as essential ones and thus produces misleading evaluation scores.1
• Although some methods prompt an LLM to directly assign weights, direct weighting requires a holistic judgment over all criteria that LLMs struggle to make reliably.1
• Extensive experiments demonstrate that ICW consistently outperforms uniform weighting and competitive weighting methods, and translates into better policy models in downstream reinforcement learning.1

Sources (1)

• [1]Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 10:05 AM

  To address these limitations, we propose Importance-Aware Criteria Weighting (ICW), which derives the weights of criteria from pairwise importance comparisons made by an LLM.

  Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally.

Extractive summary: sentences quoted from the sources.