Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models

Key points

• We ask when different ways of eliciting and reading an answer recover the same judgment, and when they measure an answer habit.1
• We collect 1.1 million replies from eight models in six languages on up to 105 moral vignettes, varying the question's form, the answer constraint, and the reading of the reply, token or sentence, in both wordings (the FAR-6 battery).1
• Story-level judgment profiles converge across the tested readouts and question forms, except for the yes/no token after a negative interrogative read under a preset mapping: on the same replies, token and sentence profiles agree at a median r = 0.67 across cells, while sentence profiles agree with free-sentence profiles at 0.94, a known linguistic variability.1
• Particle, position, and named-term preferences follow their own patterns.1

Sources (1)

• [1]Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 11:50 AM

  Multilingual evaluations often score a model's opening yes or no, a token that also carries model and language answering conventions.

  We ask when different ways of eliciting and reading an answer recover the same judgment, and when they measure an answer habit.

Extractive summary: sentences quoted from the sources.