KV cache

Timeline

1. Oct 19, 2026 · Open-source release · 1 source
   sgl-project/sglang v0.5.22
   | Model | Type | Cookbook |
2. Oct 18, 2026 · Model release · 6 sources
   deepseek-ai/DeepSeek-V4.1-Flash-1017
   deepseek-ai published the model DeepSeek-V4.1-Flash-1017 on Hugging Face.
3. Oct 18, 2026 · Opinion / analysis · 1 source
   Agens Volundr 32B v1: we fixed what you flagged in the Preview. 0/50 loops, session continuation, and an MIT engine that runs it with 262K context (measured to 239K) within a 16 GB budget
   WHAT v1 FIXES FROM THE PREVIEW
4. Oct 18, 2026 · Opinion / analysis · 1 source
   GPT-2 KV cache from scratch: correctness, latency, and memory measurements [P]
   I built GPT-2 small and a KV cache in PyTorch as a learning project, then compared the mathematical savings with measured latency and memory.
5. Oct 18, 2026 · Open-source release · 1 source
   vllm-project/vllm v0.32.0
   This release features 814 commits from 363 contributors (121 new)!
6. Oct 18, 2026 · Opinion / analysis · 1 source
   You don’t need that expensive of a build for local llms.
   Been seeing a lot of people with their 64gb of ddr5 4x3090 or whatever strata builds and it’s been sorta depressing knowing i cant afford that but i finally finished my setup today so here we go,
7. Oct 17, 2026 · Opinion / analysis · 1 source
   Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming
   I have been working on implementing adaptive KV streaming and RDNA 4 / ROCm optimizations for llama.cpp to enable running Qwen3.8-27B (IQ4XS weights, Q80/Q40 KV cache) with its full 262,144-token context window on a single 16 GB RX 9070 XT GPU at 80+ t/s decode.
8. Oct 17, 2026 · Opinion / analysis · 1 source
   Estimating the Theoretical Parameter Capacity of a Hypothetical GPT-6-Class Model [D]
   I've been exploring a hypothetical question: How large could a model's parameter count be if a major AI provider allocated a substantial GPU cluster to inference while targeting 66 output tokens per second per active user?
9. Oct 17, 2026 · Opinion / analysis · 1 source
   model : add Kolibri-1 (Aleph-Alpha) support by riteme · Pull Request #30673 · ggml-org/llama.cpp
   Kolibri is Aleph Alpha's mixture-of-experts (MoE) reasoning model, with a focus on German and English.
10. Oct 16, 2026 · Open-source release · 5 sources
    Welcome Beam: a 501B open model you can own, customize, and scale
    Beam is focused on coding and agentic use cases.
11. Oct 16, 2026 · Opinion / analysis · 1 source
    Improved Qwen3.8-Flash Chat Template (w/ Sharp Template terse option)
    I've been maintaining my own chat template for Qwen3.8-Flash-Next since it released, and after a discussion in another thread I decided to share it.
12. Oct 16, 2026 · Research paper · 1 source
    FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
    We present FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs.
13. Oct 16, 2026 · Research paper · 2 sources
    DLite: History-free Speculative Decoding for Agentic AI
    In this paper, we identify the growing historical context as a common source of both problems and introduce DLite, a history-free speculative decoding framework for agentic AI.
14. Oct 16, 2026 · Research paper · 1 source
    CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective
    Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard autoregressive key--value cache.
15. Oct 16, 2026 · Benchmark result · 1 source
    Qwen Flash Next potential SOTA Q2 nobody talks about
    Also enhanced loop resitance is advertised.
16. Oct 16, 2026 · Research paper · 1 source
    Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference
    Long-context decoding in large language models (LLMs) is increasingly bottlenecked by the memory and computation required to attend over an ever-growing Key-Value (KV) cache at each autoregressive step.
17. Oct 15, 2026 · Research paper · 1 source
    First Passage Attention
    Augmenting softmax attention with forget gates typically relies on summing decay contributions along the key--query path.
18. Oct 15, 2026 · Research paper · 1 source
    SAFETYEDIT: Sparse Internal Targets with Prompt-Wide Control for LLM Jailbreak Backdoors
    Model-editing backdoors aim to induce harmful responses under a trigger while preserving ordinary behavior on untriggered input.
19. Oct 15, 2026 · Model release · 2 sources
    Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint
    Specifically, Granite-Switch-4.1-8B and -30B models are configured to include:
20. Oct 15, 2026 · Open-source release · 1 source
    huggingface/transformers v5.20.0: Release v5.20.0
    Muse Spark Open is a multimodal mixture-of-experts model from Meta that takes text, images, audio, and video as input and generates text.
21. Oct 15, 2026 · Research paper · 1 source
    DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction
    Key-value (KV) cache management has become a critical bottleneck for scaling Large Language Models (LLMs) to long-context scenarios due to linear memory growth.
22. Oct 15, 2026 · Research paper · 1 source
    Augment the Cache, Replace the Activation: A 2x2 Study of Internal-State Sharing
    Multi-agent systems built from frozen language models communicate through text, which discards the internal state the sender already computed and makes the receiver pay to rebuild it.
23. Oct 15, 2026 · Research paper · 1 source
    Close Enough is Nowhere Near: Fidelity-Consumability Decoupling in KV Caches
    We show that geometric similarity to that cache does not predict whether the reader can use it, and the failure is not gradual, so a more faithful substitute is not the remedy it appears to be.
24. Oct 15, 2026 · Research paper · 2 sources
    When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
    Reusing the attention KV cache across requests is a standard way to cut prefill cost in long-context LLM serving, and production systems increasingly back the per-engine cache with a remote tier shared by every engine.
25. Oct 15, 2026 · Research paper · 1 source
    PriorityFEC: Deadline-Aware Streaming FEC for KV-Cache Handoff in Edge Multi-Agent Systems
    Multi-agent large language model (LLM) systems are increasingly deployed at the edge to support complex tasks that require collaboration and information exchange among multiple agents.
26. Oct 15, 2026 · Research paper · 1 source
    The Density of Thought: Predicting and Steering Reasoning in Large Language Models
    We investigate whether variation in information density across chain-of-thought (CoT) reasoning steps predicts answer correctness and can serve as a controllable inference-time target.
27. Oct 15, 2026 · Research paper · 1 source
    From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models
    This survey reframes long-context compression as a memory-management problem for allocating inference-time resources.
28. Oct 15, 2026 · Opinion / analysis · 1 source
    4x R9700 builds with $127 used EPYCs: $10.7k all-in per machine. I built three (384GB VRAM)
    I saw the 4x R9700 build for a 10-person startup here a few days ago and wanted to share my results, since I opted to spend most of the money on the cards.
29. Oct 15, 2026 · Research paper · 1 source
    Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agents
    To address this challenge, we introduce Selection-based Structured Reasoning (SSR), a framework that reformulates reasoning as selection instead of open-ended generation.
30. Oct 14, 2026 · Open-source release · 4 sources
    Fish Audio Drama 3 Technical Report
    We introduce Fish Audio Drama 3, an open-source speech generation model that turns text into directed performances, built on a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token.

Often appears with