Language-model ratings of depression reflect the rater more than the patient

Key points

• Depression has no diagnostic blood test.1
• We pre-registered 880 language-model raters, crossing 11 open models with prompting and scoring choices, and applied them to 189 interviews against the eight-item Patient Health Questionnaire.1
• Model choice explained 30.0% of summed-symptom score variance, stable participant differences 10.5%.1
• Two randomly drawn raters with area under the receiver operating characteristic curve (AUC) >= 0.70 disagreed on screening decisions for 40% of participants, on average.1

Sources (1)

• [1]Language-model ratings of depression reflect the rater more than the patient
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 03:08 PM

  Language models promise tireless, consistent assessment, but can accurate raters disagree about individuals?

  Depression has no diagnostic blood test.

Extractive summary: sentences quoted from the sources.