Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming

Key points

• An LLM attacker edits a transcript so that the agent causes a specified harm, such as leaking an API key or spawning an unmonitored agent.1
• The edited actions are never executed, so we use an LLM verifier to reject attacks that are inconsistent with the original transcript or below a severity bar.1
• We applied Screwtape to 94 Claude Code and Codex transcripts from Redwood Research staff, optimizing against our own monitors.1
• We therefore demonstrate that a red-teamer without access to a developer's infrastructure can find confirmed failures of the developer's monitor.1

Sources (1)

• [1]Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 09:08 PM

  We present Screwtape, a pipeline that converts real agent transcripts into attacks that evade action-only monitors.

  An LLM attacker edits a transcript so that the agent causes a specified harm, such as leaking an API key or spawning an unmonitored agent.

Extractive summary: sentences quoted from the sources.