Decouple to Forget: Semantic Mediation for Sensitive Unlearning in Large Language Models

Key points

• Machine unlearning for large language models seeks to remove specified knowledge while preserving general capability, yet current methods often treat an entire response as the unit of forgetting.1
• For each example to be forgotten, D2F extracts a fact-level "mediator" representation from the answer, identifies which parts are sensitive, and removes only those parts while leaving the rest of the structure intact.1
• We then generate counterfactual answers grounded in this mediator that preserve non-sensitive content but exclude sensitive facts, employing preference-based contrastive optimization to reduce the likelihood of information leakage while maintaining helpfulness and fluency.1
• Experiments across various benchmark datasets show that D2F achieves a better balance between forgetting and retention than strong baseline methods.1

Sources (1)

• [1]Decouple to Forget: Semantic Mediation for Sensitive Unlearning in Large Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 03:32 AM

  In this paper, we introduce Decouple to Forget (D2F), a targeted unlearning method inspired by semantic mediation that separates sensitive factual content from non-sensitive meaning and surface expression.

  Machine unlearning for large language models seeks to remove specified knowledge while preserving general capability, yet current methods often treat an entire response as the unit of forgetting.

Extractive summary: sentences quoted from the sources.