FairStress: A deeper evaluation of fairness in LLM decision-making
Today we’re introducing FairStress, a benchmark for measuring demographic fairness in LLMs beyond the choice a model ultimately makes in decisions impacting people.
ProofVendor claim only
Key points
- FairStress is available for download, and we’re also building it into Olmo-eval, our open workbench for running and comparing model evaluations.
- Standard demographic fairness benchmarks often test whether a model selects the same candidate in otherwise identical scenarios after the candidates’ demographic identities are swapped.
- Across open-weight and frontier models including Olmo, Llama, Qwen, Gemma, Claude, and the GPT family, and across multiple stages of training, we find that the same persuasive arguments – which we call adversarial prefixes – can make models more or less likely to change their decisions depending on which demographic group the argument favors.
- With FairStress, our target is symmetric fairness—when the evidence supports a decision, a model should apply the same criteria and be equally resistant to changing that decision regardless of which demographic group benefits.
Sources (1)
- [1]FairStress: A deeper evaluation of fairness in LLM decision-makingHugging Face Blog · Oct 20, 04:09 PM
Today we’re introducing FairStress, a benchmark for measuring demographic fairness in LLMs beyond the choice a model ultimately makes in decisions impacting people.
FairStress is available for download, and we’re also building it into Olmo-eval, our open workbench for running and comparing model evaluations.
Extractive summary: sentences quoted from the sources.
Hype check
Independently evaluated5 independent looks since
- Evaluation · Oct 20, 2026OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants
- Evaluation · Oct 20, 2026Agnes-AI/Agnes-3.0-Qwen (QFN finetune)
- Evaluation · Oct 20, 2026Evolving Semantic Indexes for Generative Retrieval
- Evaluation · Oct 20, 2026SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models
- Evaluation · Oct 20, 2026SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges
- Follow-up · Oct 20, 2026GPT-6-Astra Understands Touch and Outperform Vision--Tactile--Language--Action Models
Before this
- Oct 20, 2026Hallucinations of Large Language Models in Cross-Language Code Translation: Phenomena and Mitigation
- Oct 20, 2026MiniMax H3 - Really is the best open-source video model.
- Oct 20, 2026Anthropic and Citrini Use Claude to Design Scorpion Neurotoxin Binders
- Oct 19, 2026EGI-BENCH: An Enterprise General Intelligence Benchmark Suite
- Oct 19, 2026Introducing Clairic, a new System One class of extraction models
- Oct 19, 2026Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM