WUSPruner: Second-Moment-Optimal Transforms for Structured Pruning

Key points

• Structured pruning is typically formulated as identifying which model dimensions to remove, often within the model's original representation.1
• We show that under positive-definite second moments, WUS transform minimizes the local reconstruction loss over all invertible changes of basis.1
• Guided by this result, we develop a scalable pruning method based on blockwise WUS transformations and compact input projections.1
• Experiments on Llama and Qwen models from 8B to 72B parameters show consistent improvements over structured pruning baselines across language modeling, knowledge, commonsense reasoning, mathematics, and code generation benchmarks, while also delivering practical reductions in inference latency and memory usage.1

Sources (1)

• [1]WUSPruner: Second-Moment-Optimal Transforms for Structured Pruning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 06:47 AM

  We introduce WUSPruner, a transform-based framework for structured pruning of large language models.

  Structured pruning is typically formulated as identifying which model dimensions to remove, often within the model's original representation.

Extractive summary: sentences quoted from the sources.