TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference
Long-context large language model (LLM) inference is important for modern intelligent applications but remains constrained by peak memory during prompt prefill.
ProofPaper ↗
Key points
- Existing KV cache compression reduces decoding memory but typically requires full-prompt prefill to preserve global context, retaining a high construction peak.
- Online chunk-level eviction lowers memory earlier but selects before each layer has observed the complete prompt, potentially degrading cache construction and decoding.
- We present TideKV, an on-GPU cache-construction architecture using layer-major delayed eviction.
- Experiments across multiple computing platforms show that TideKV preserves the inference quality and final cache size of the evaluated compression methods.
Sources (1)
- [1]TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM InferencearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 06:44 AM
Long-context large language model (LLM) inference is important for modern intelligent applications but remains constrained by peak memory during prompt prefill.
Existing KV cache compression reduces decoding memory but typically requires full-prompt prefill to preserve global context, retaining a high construction peak.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 17, 2026deepseek-ai/DeepSeek-V4.1-Flash-1017
- Oct 16, 2026Hugging Face ❤️ TorchTPU: TPUs, Natively
- Oct 16, 2026Welcome Beam: a 501B open model you can own, customize, and scale
- Oct 16, 2026DLite: History-free Speculative Decoding for Agentic AI
- Oct 15, 2026Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint
- Oct 15, 2026PriorityFEC: Deadline-Aware Streaming FEC for KV-Cache Handoff in Edge Multi-Agent Systems