The Fragility and Dynamics of LLM Pruning-Recovery Steps

Key points

• Structural pruning followed by a recovery stage is the standard recipe for compressing large language models (LLMs), yet the two stages are typically designed and tuned in isolation.1
• In this work, we study how their interaction shapes the compressed model, sweeping ordering, the data that drives each stage, and the alignment of their training objectives.1
• We find in our setting that in-domain data is the strongest pruning signal, whereas instruction-style data recovers the most.1
• We further analyze how the importance ranking drifts before and after pruning.1

Sources (1)

• [1]The Fragility and Dynamics of LLM Pruning-Recovery Steps
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 12:38 PM

  We find that instruction-following is the first capability to collapse under pruning, well before knowledge, reasoning, or core language modeling, and that much of the pruning in-domain loss is likely not knowledge loss but a domain-agnostic failure to respond coherently to the prompt.

  Structural pruning followed by a recovery stage is the standard recipe for compressing large language models (LLMs), yet the two stages are typically designed and tuned in isolation.

Extractive summary: sentences quoted from the sources.