The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check

Key points

• We hold the target model, the prompts and the responses fixed and change only the grader.1
• Three dense graders of 8B and 9B parameters, from three model families, move compliant responses to the neighbouring class, evasion, instead of scattering them across classes.1
• The collapse decides how the rate behaves under a change of grader.1
• We propose a reporting minimum that keeps the compliance rate and adds its per-class label counts and the named grader.1

Sources (1)

• [1]The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 09:20 PM

  Jailbreak papers report an attack success rate, the share of responses that deliver the harmful request, and read it as a property of the target model.

  We hold the target model, the prompts and the responses fixed and change only the grader.

Extractive summary: sentences quoted from the sources.