Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers

Key points

• TypeSafe AI's Jev is a new kind of safety judge.1
• We build a test harness that sends every system the same payload under the same policy, and run it on four public-data tasks, from rule-based ones (input moderation, injection detection, routing) to a judgment-based one (phishing detection), with fifty attack variants from thirteen families.1
• We find that Jev is the most robust system on input moderation: eight forgeries of its structured input bypass none of its decisions, and a sentence asking only why someone might make a harmful request bypasses 9% of its blocks, against 56% for GPT-6 Astra.1
• The local classifiers fail differently: a training-example statement makes Llama Guard 3 miss 62% of the phishing it had flagged.1

Sources (1)

• [1]Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 06:43 AM

  In this paper, we present the first controlled adversarial evaluation of Jev.

  TypeSafe AI's Jev is a new kind of safety judge.

Extractive summary: sentences quoted from the sources.