Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming

Key points

• We propose LayerTrim, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth.1
• LayerTrim is the first training-free structured pruning method designed to preserve long-form reasoning without gradients or recovery training.1
• On Qwen3-14B, LayerTrim achieves 1.24$\times$ faster prefill and 19.7% lower peak GPU memory than the dense model, while better preserving reasoning ability.1
• These results demonstrate that LayerTrim improves reasoning efficiency through both structural compression and reduced token generation.1

Sources (2)

• [1]Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 07:55 PM

  Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models.

  We propose LayerTrim, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth.

• [2]Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming
  Hugging Face Daily Papers · Oct 13, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.