Not All Change Is Damage: Update-Matched Unlearning for Large Language Models

Key points

• For models that continue to be fine-tuned after release, this goal is insufficient: they must also resist relearning the deleted knowledge while preserving plasticity, the ability to learn new tasks.1
• We therefore formulate the reference-timing hypothesis that, at comparable target suppression, an update-matched reference better preserves plasticity than a pre-update one.1
• To test and address this hypothesis, we propose Update-Matched Unlearning (UMU), which applies the same simulated relearning update to the defended model and to an undefended reference initialized from the same unlearned model, and uses new-task learning to guide model selection.1
• When the reference does not receive the update, relearning remains similarly low but new-task accuracy falls below the base method, supporting the reference-timing hypothesis and showing that robust unlearning should prevent recovery selectively without suppressing learning itself.1

Sources (1)

• [1]Not All Change Is Damage: Update-Matched Unlearning for Large Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 04:19 AM

  Unlearning aims to remove selected knowledge from large language models to mitigate privacy and safety risks while preserving useful capabilities.

  For models that continue to be fine-tuned after release, this goal is insufficient: they must also resist relearning the deleted knowledge while preserving plasticity, the ability to learn new tasks.

Extractive summary: sentences quoted from the sources.