The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction
Offline compaction of the key--value cache (KV-cache) reduces the cost of repeatedly querying a fixed context by constructing a smaller cache once and reusing it subsequently.
ProofPaper ↗
Key points
- The central challenge of compaction is deciding which cache entries to delete and which to retain.
- In this work, we introduce KV-Surgeon, which uses calibration data to take interactions into account in a tractable way.
- We derive a quadratic surrogate that admits closed-form expressions for jointly making the optimal greedy removal decision as well as compensation within the remaining entries.
- At $50\times$ compaction, KV-Surgeon improves question-answering accuracy by up to four points over attention matching (Zweiger et al., 2026), the state of the art, at comparable cost.
Sources (1)
- [1]The KV-Surgeon: Compensation-Aware Offline KV-Cache CompactionarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 07:31 AM
Offline compaction of the key--value cache (KV-cache) reduces the cost of repeatedly querying a fixed context by constructing a smaller cache once and reusing it subsequently.
The central challenge of compaction is deciding which cache entries to delete and which to retain.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026TorchNeuron: running HF models on AWS Trainium is easier than ever
- Oct 19, 2026MemT-1.0: Towards Memory-Native Evolution
- Oct 17, 2026model : add Kolibri-1 (Aleph-Alpha) support by riteme · Pull Request #30673 · ggml-org/llama.cpp
- Oct 17, 2026deepseek-ai/DeepSeek-V4.1-Flash-1017
- Oct 17, 2026Nemotron 3.5 Bolt, Muse Spark Open, and a Fast LFM2.5 24B
- Oct 16, 2026Hugging Face ❤️ TorchTPU: TPUs, Natively