Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks

Key points

• Yet each additional call gives the deployed system another opportunity to detect the attack and intervene.1
• We introduce $ASR{STEP}$, which penalizes additional victim-model calls while retaining binary success judgments.1
• Our evaluation considers GCG \cite{zou2023universal}, PAIR \cite{chao2025jailbreaking} with a step budget, and a single-step PAP \cite{zeng2024johnny} baseline on JailBreakBench across three model families.1
• While for PAIR \cite{chao2025jailbreaking}, we see a wider discounting range between 3.4-12 percentage points, showing that the "step" quantity differs per iterative strategy.1

Sources (1)

• [1]Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:36 PM

  Attack success rate (ASR) evaluates jailbreaks against large language models without accounting for repeated calls to the victim model.

  Yet each additional call gives the deployed system another opportunity to detect the attack and intervene.

Extractive summary: sentences quoted from the sources.