MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization

Key points

• Existing KV cache quantization methods have explored different compression granularities, including layer-wise precision adjustment and token-level information preservation.1
• In practical deployment, however, the key challenge is not merely compression, but how to allocate limited quantization resources among heterogeneous precision preservation strategies under a fixed KV cache memory budget.1
• We propose MultiAlloc, a profiling-guided precision allocation framework for KV cache quantization that adaptively distributes limited quantization resources between layer-wise precision enhancement and token-wise preservation.1
• Extensive experiments on reasoning and long-context benchmarks demonstrate that MultiAlloc consistently improves over layer-wise, token-wise, and recent multi-granularity KV cache quantization methods under matched memory budgets across diverse models, including Mistral, Llama, DeepSeek-Qwen, and QwQ.1

Sources (1)

• [1]MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 04:10 PM

  The Key-Value (KV) cache constitutes a dominant source of memory consumption during large language model (LLM) inference, particularly in long-context and complex reasoning scenarios.

  Existing KV cache quantization methods have explored different compression granularities, including layer-wise precision adjustment and token-level information preservation.

Extractive summary: sentences quoted from the sources.