Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring
Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs.
ProofPaper ↗
Key points
- To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design.
- We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation.
- Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting.
- We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales.
Sources (1)
- [1]Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text ScoringarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 04:42 PM
Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs.
To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026Decomposing Brain-LLM Alignment with Interpretable Semantic Representations
- Oct 15, 2026Incentivizing Tool-Use Reinforcement Learning for Post-Arrival Emergency Care Prediction
- Oct 15, 2026Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference
- Oct 15, 2026Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries
- Oct 15, 2026Training Reasoning Models to Generate Diverse Scientific Ideas
- Oct 14, 2026Optimal Hydra: Model Heterogeneous Federated Learning for LLMs using Data-Informed Hydra-Nets
