Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

Key points

• Therefore, we propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings.1
• To test our meta-evaluation framework, we developed a large-scale multi-lingual style-personalized text generation (SPTG) benchmark, collecting multiple annotations across 10 topologically diverse languages and four evaluation dimensions, each with varying degrees of subjectivity.1
• We meta-evaluated 32 metrics from three style evaluation paradigms (string-based, embedding-based, and LLM-judges) and their ensembles over these 40 evaluation settings.1
• Our analysis revealed that even simple string- and embedding-based measurements can match strong LLM-judges when calibrated to human ratings.1

Sources (1)

• [1]Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 08:31 PM

  Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations.

  Therefore, we propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings.

Extractive summary: sentences quoted from the sources.