Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

Key points

• Decision models are a new class of language models that return a probability for each point of a rating scale instead of text.1
• We ask whether these models, or the newest generative LLMs, can stand in for human respondents on a psychometric survey.1
• We test fourteen decision models from eleven providers and 47 generative LLMs on an open Lithuanian organisational-psychology survey (n=263, 68 items), with 100 real respondent profiles, the Psychometric Similarity Score (PSS) and statistical baselines.1
• In 5 of 10 product lines the newest generative release scores below an earlier release of the same line beyond resampling noise, and none gains beyond it.1

Sources (2)

• [1]Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents
  Hugging Face Daily Papers · Oct 10, 12:00 AM

  Language models are increasingly used to create synthetic survey samples, also called silicon samples.

  Decision models are a new class of language models that return a probability for each point of a rating scale instead of text.

• [2]Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 01:34 PM · same content

Extractive summary: sentences quoted from the sources.