Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates
Jailbreak methods are commonly compared using Attack Success Rate (ASR), but these comparisons depend on how responses are generated and judged.
ProofPaper ↗
Key points
- In this paper, we examine eight attacks against three open-weight language models using three evaluation procedures.
- Low-resource language attacks achieve a mean ASR of 96.0% under keyword matching but 2.7% under a Qwen3 evaluator following StrongREJECT scoring instructions.
- Changing the number and arrangement of attempts reverses the ranking of DeepInception and rewriting.
- Beyond evaluator sensitivity, these findings raise a safety concern: repeated sampling can expose residual unsafe behavior despite an initial refusal, suggesting probabilistic suppression rather than robust exclusion.
Sources (1)
- [1]Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success RatesarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:39 PM
Jailbreak methods are commonly compared using Attack Success Rate (ASR), but these comparisons depend on how responses are generated and judged.
In this paper, we examine eight attacks against three open-weight language models using three evaluation procedures.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026Learning to Couple: Context-Adaptive Joint Denoising for Diffusion Language Models
- Oct 19, 2026TorchNeuron: running HF models on AWS Trainium is easier than ever
- Oct 19, 2026TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
- Oct 18, 2026Dual-Slot: From Serial Loops to Parallel Slots in Language Models
- Oct 17, 2026Nemotron 3.5 Bolt, Muse Spark Open, and a Fast LFM2.5 24B
- Oct 16, 2026[Research] 2,788 tokens/s on a single RTX 5090 with a diffusion LLM — Introducing ORL (fully open source)