Continual Concept Personalization in Diffusion Models via Residual-Space Adaptation

Key points

• In practical applications, concepts often arrive sequentially, requiring models to continually learn new subjects, objects, or styles while minimizing interference with previously acquired concepts.1
• We address this continual concept personalization setting by extending residual-space adaptation to the stochastic adaptation process of pretrained diffusion models.1
• Beyond these empirical gains, the learned residual-space dynamics reveal where adaptation capacity is concentrated within the diffusion model.1
• Restricting adaptation to the identified layers by our method preserves nearly the same performance while reducing the number of trainable parameters by up to 72%, demonstrating that residual-space dynamics can identify the layers that contribute most to continual concept personalization and directly guide more parameter-efficient adaptation.1

Sources (1)

• [1]Continual Concept Personalization in Diffusion Models via Residual-Space Adaptation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 05:26 PM

  Text-to-image personalization aims to equip pretrained diffusion models with the ability to learn user-specific concepts from only a few reference images.

  In practical applications, concepts often arrive sequentially, requiring models to continually learn new subjects, objects, or styles while minimizing interference with previously acquired concepts.

Extractive summary: sentences quoted from the sources.