Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too

Key points

• We study novel connections between downstream performance of compressed models and the ability to retain gradient information via influence-function preservation.1
• We introduce $CACTUS$, a novel ADMM-based optimizer supporting both pruning and sparse-plus-low-rank decompositions of pre-trained weights.1
• At $50%$ pruning of Qwen3-4B, $CACTUS$ reduces the C4 perplexity gap to the dense model by $45%$, improves compressed-to-dense per-sample gradient cosine similarity on WikiText-2 from $0.670$ to $0.755$, and runs over $28\times$ faster than LLM-Surgeon on substantially weaker hardware.1
• Finally, integrating $CACTUS$ with LoGRA, a recent gradient-projection strategy, on Qwen3-4B with MLP layers pruned to 2:4 sparsity and a rank-64 correction retains a $0.711$ cosine alignment with dense LoGRA-projected C4 gradients.1

Sources (1)

• [1]Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 03:09 PM

  Post-training pruning enables efficient deployment, but conventional methods do not explicitly preserve internal-gradient alignment between sparse and dense models.

  We study novel connections between downstream performance of compressed models and the ability to retain gradient information via influence-function preservation.

Extractive summary: sentences quoted from the sources.