Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On

Key points

• It is not: it is a property of a policy, and the policy enters the model through its labels.1
• First, the guardrail's own training corpus already contained the benchmarks: JailbreakBench-benign is 88% and OR-Bench-hard 82% present in the training pools, two over-refusal benchmarks, the kind a guardrail reports to argue it does not over-block.1
• Second, safety benchmarks stratify sharply by how much of their score is decided by the label definition rather than by the model: with architecture and optimiser fixed, the range across the ten arms is 0.99 on HarmBench-copyright and 0.043 on WildGuard-benign, a 23x spread.1
• For a distilled student the teacher's soft labels do this even when every hard label carries the other policy, so teacher provenance is an unrecorded confound in any released checkpoint.1

Sources (2)

• [1]Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On
  Hugging Face Daily Papers · Oct 18, 12:00 AM

  A guardrail classifier is reported as if unsafe were a property of text.

  It is not: it is a property of a policy, and the policy enters the model through its labels.

• [2]Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 07:02 AM · same content

Extractive summary: sentences quoted from the sources.