Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages
We propose predicting disagreement from interpretable semantic properties of the text.
ProofPaper ↗
Key points
- Annotator disagreement is increasingly recognized as a meaningful property of subjective tasks, rather than merely noise to be removed.
- Predicting which items are likely to be contested could help allocate annotation effort and route difficult cases to human review.
- An LLM answers a set of binary questions covering harmful-content characteristics, disagreement-related cues such as borderline severity and missing context, and language-sensitive cues such as local slurs.
- A classifier predicts disagreement from these answers, and we analyze which questions contribute most to the predictions.
Sources (1)
- [1]Decomposing and Predicting Annotation Disagreement in Harmful Content Across LanguagesarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 03:49 PM
We propose predicting disagreement from interpretable semantic properties of the text.
Annotator disagreement is increasingly recognized as a meaningful property of subjective tasks, rather than merely noise to be removed.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards
- Oct 13, 2026Copy-Paste Language Models
- Oct 13, 2026Beyond the Verdict: Agentic Code Judging with Verifiable Feedback
- Oct 12, 2026Redesigning source control at Anthropic
- Oct 12, 2026Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards
- Oct 12, 2026LLM-to-Flow Distillation: Teaching Natural Language Goals to Scientific Generators
