Defending Against Dormant Poisoning Attacks Across Language and Multimodal Agents

Key points

• We show two agentic risks: (1) harmful behaviors activated by benign fine-tuning can extend to unseen actions; (2) multiple harmful behaviors can coexist and be jointly activated by benign downstream fine-tuning.1
• We exploit a key property of dormant poisoning: the attacker must preserve safe behavior at release to conceal poisoning, and this behavior can serve as a safety reference.1
• Leveraging this property, we automatically construct a Safety Buffer pairing harmful inputs with the released model's responses and benign versions generated by the same model to remove harmful intent.1
• To address this limitation, we introduce LookAhead Defense, which previews benign fine-tuning updates using the Safety Buffer and penalizes only updates predicted to weaken safe behavior on harmful inputs or increase the likelihood of refusals on benign versions in the Safety Buffer.1

Sources (1)

• [1]Defending Against Dormant Poisoning Attacks Across Language and Multimodal Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 02:26 AM

  Recent work shows that attackers can implant dormant malicious behaviors that remain hidden at release but emerge after benign fine-tuning.

  We show two agentic risks: (1) harmful behaviors activated by benign fine-tuning can extend to unseen actions; (2) multiple harmful behaviors can coexist and be jointly activated by benign downstream fine-tuning.

Extractive summary: sentences quoted from the sources.