AuthGateBench: Benchmarking LLMs as Authorization Gate

Key points

• Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classifier, typically a large language model (LLM).1
• Such systems, adopted in tools like Claude Code and Codex, remain vulnerable to failure modes such as scope escalation, where the agent takes actions exceeding what the user authorized.1
• Existing benchmarks focus mainly on malicious requests, evaluating refusal under an implicit policy.1
• We implement a novel pipeline where an LLM generates contrast pairs with identical action and trajectory but differing request, flipping the label between block and allow.1

Sources (1)

• [1]AuthGateBench: Benchmarking LLMs as Authorization Gate
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 08:55 AM

  We introduce AuthGateBench, the first policy-guided benchmark for authorization gating under benign requests.

  Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classifier, typically a large language model (LLM).

Extractive summary: sentences quoted from the sources.