Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization

Key points

• On-Policy Self-distillation is a self-improvement paradigm that enables large language models (LLMs) to improve their capabilities by leveraging the same model as both teacher and student under different conditioning contexts.1
• However, we find that OPSD methods can progressively degrade previously acquired capabilities: queries that are correctly responded at earlier training stages may become incorrectly responded as optimization proceeds, even the overall performance continues to improve.1
• Further analysis reveals that this degradation coincides with a deterioration of the geometry of hidden representations, characterized by reduced intra-cluster compactness and diminished inter-cluster separation.1
• Extensive experiments on multiple OPSD methods and benchmarks show that RGR generally alleviates capability degradation while retaining capability improvements.1

Sources (1)

• [1]Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 01:28 AM

  Motivated by this observation, we propose Representation Geometry Regularization (RGR), which preserves representation geometry during post-training by encouraging compactness within cluster and separation across cluster.

  On-Policy Self-distillation is a self-improvement paradigm that enables large language models (LLMs) to improve their capabilities by leveraging the same model as both teacher and student under different conditioning contexts.

Extractive summary: sentences quoted from the sources.