Same Operation, Different Permission: Evaluating Whether Guardrails Honor Authorization

Key points

• As language-model agents write code, move money and change records on behalf of users, deciding whether an action is authorized has become a central safety problem.1
• Guardrails are often placed in front of these actions, yet few evaluations ask whether their decisions follow authorization: blocking unauthorized actions while still allowing the same actions under valid permission.1
• It combines expert-annotated natural-language scenarios with executed and verified operations, plus externally sourced native-policy requests. \bench{} checks each decision for correctness, usability and resulting effect, and tracks each request across guardrail changes.1
• Our experiments show that many blocked operations stem from failed reviews, not correct denials.1

Sources (1)

• [1]Same Operation, Different Permission: Evaluating Whether Guardrails Honor Authorization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 02:17 PM

  We introduce \bench{}, a benchmark that contrasts authorization evidence for the same operation in its language track.

  As language-model agents write code, move money and change records on behalf of users, deciding whether an action is authorized has become a central safety problem.

Extractive summary: sentences quoted from the sources.