Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation

Key points

• Such a comparison assumes Script Invariance: the score should not depend on the writing system that carries the target.1
• We test it on IndicMT Eval by re-encoding the target into Latin script, which changes orthographic form while holding content and human ratings fixed.1
• Script identity then accounts for 22.9% of native-script COMET variance, and agreement with annotators falls in all five languages studied.1
• We trace the effect to the tokeniser and measure it with three label-free diagnostics.1

Sources (1)

• [1]Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 11:11 AM

  COMET reports translation quality as a single number, and that number is routinely compared across target languages written in different scripts.

  Such a comparison assumes Script Invariance: the score should not depend on the writing system that carries the target.

Extractive summary: sentences quoted from the sources.