MOSAIC: Multi-Objective Search for Allocating Inference Cache Across LLM Decoder Layers

Key points

• Token eviction methods such as SnapKV, H2O, AdaKV and L2Norm shrink it, but they give every transformer layer the same budget, although layers differ in which tokens they need.1
• We present MOSAIC, a multi-objective neural architecture search that learns per-layer KV-cache budgets for any eviction method.1
• Across four eviction methods on Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2, MOSAIC beats uniform allocation in 146 of 154 (task, method, budget) settings, by up to +48 points on RULER and +11.9 on LongBench (LCC).1
• The learned allocations consistently favour a small set of middle layers.1

Sources (1)

• [1]MOSAIC: Multi-Objective Search for Allocating Inference Cache Across LLM Decoder Layers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 05:51 PM

  The Key-Value (KV) cache is the main memory bottleneck of long-context LLM inference.

  Token eviction methods such as SnapKV, H2O, AdaKV and L2Norm shrink it, but they give every transformer layer the same budget, although layers differ in which tokens they need.

Extractive summary: sentences quoted from the sources.