Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM

Key points

• vLLM is an open-source inference and serving engine for large language models (LLMs).1
• One of the ways vLLM performs this transfer is through the NVIDIA Inference Xfer Library (NIXL) connector, which moves KV blocks over interconnects that support Remote Direct Memory Access (RDMA), such as AWS Elastic Fabric Adapter (AWS EFA, the low-latency RDMA network used across servers in AWS).1
• This post introduces a push-based NIXL connector that moves post-prefill transfer coordination and setup off the critical path.1
• If you run disaggregated prefill/decode inference with vLLM, push mode can improve application responsiveness by reducing TTFT and TPOT, particularly at high request rates.1

Sources (1)

• [1]Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
  AWS Machine Learning Blog · Oct 19, 04:05 PM

  LLM inference has two distinct phases: prefill, which processes the input prompt and builds the key-value (KV) cache, and decode, which uses that cache to generate output one token at a time.

  vLLM is an open-source inference and serving engine for large language models (LLMs).

Extractive summary: sentences quoted from the sources.