"You" Is Not Toxic: How LLM Labelers Rewrite the Web's Conversation Laws
What the Web community believes about toxic conversations, such as that toxicity rises in longer and deeper threads, was measured mostly with Google's Perspective API, which is shutting down; large language models (LLMs) will make the next measurements.
ProofPaper ↗
Key points
- We find that LLM labelers mistake second-person address for toxicity, and that this one bias manufactures laws that human labels do not show.
- On 1.25 million news comments with crowd labels, Claude Haiku flags 12% of the non-toxic comments that contain no "you" and 51% of those with four or more; a single crowd rater flags 6% and 10%.
- Deeper replies address someone more often, so under Haiku toxicity rises with reply depth (+0.93 percentage points per level), while under the crowd's labels it slightly declines (-0.12).
- Agreement with human labels does not reveal the bias: we prove that a fixed Cohen's kappa leaves a law nearly unconstrained.
Sources (1)
- [1]"You" Is Not Toxic: How LLM Labelers Rewrite the Web's Conversation LawsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 01:29 PM
What the Web community believes about toxic conversations, such as that toxicity rises in longer and deeper threads, was measured mostly with Google's Perspective API, which is shutting down; large language models (LLMs) will make the next measurements.
We find that LLM labelers mistake second-person address for toxicity, and that this one bias manufactures laws that human labels do not show.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026Characterising and Analysing Retrieval-Augmented Generation Techniques
- Oct 15, 2026NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning
- Oct 15, 2026[AINews] Modal’s $1B ARR, $16B Series D makes it NYC’s newest AI Infra Decacorn, 3x usage and 3x valuation in 5 months, 15x in 1 yr
- Oct 15, 2026Decomposing Brain-LLM Alignment with Interpretable Semantic Representations
- Oct 15, 2026Incentivizing Tool-Use Reinforcement Learning for Post-Arrival Emergency Care Prediction
- Oct 15, 2026Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries