ResearchResearch paperSafety & Alignment · Reasoning & Planning · Large Language Models1 source · Oct 20, 2026

SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges

We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators as reusable agent instructions.

LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training.
Existing methods for reliability testing perturb inputs with known expected effects on judge scores.
Prominent LLM-judge testing harnesses combine human-authored perturbation recipes with synthetic test generation, leaving scorer-specific recipe ideation outside the automated pipeline.
On faithfulness, SANE achieves strong human alignment in judge ranking: its test-suite scores rank 11 judge models consistently with human-labeled FaithBench performance (Spearman rho = 0.852), while JRH does not (rho = 0.100).

[1]SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 08:04 PM
We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators as reusable agent instructions.
LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training.

Extractive summary: sentences quoted from the sources.

Before this