Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Key points

• Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation.1
• In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators.1
• Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation.1
• To this end, we introduce $CaLLM$, a post hoc framework that multicalibrates an LLM judge, thereby bounding all such biases simultaneously.2

Sources (2)

• [1]Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
  Apple Machine Learning Research · Oct 12, 12:00 AM

  We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains.

  Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation.

• [2]A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 10:23 AM

  To this end, we introduce $CaLLM$, a post hoc framework that multicalibrates an LLM judge, thereby bounding all such biases simultaneously.

Extractive summary: sentences quoted from the sources.