Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets

Key points

• This open-source, Kubernetes-native layer turns vLLM or SGLang model servers into one coordinated pool.1
• A serving stack: Google reports a published benchmark on a 17,000-node multi-region GKE deployment.1
• Runs on: Kubernetes with NVIDIA, AMD or Intel GPUs or Google TPUs, over vLLM or SGLang.1
• llm-d was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA.1

Sources (1)

• [1]Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets
  MarkTechPost · Oct 21, 04:33 AM

  Google team have released llm-d 1.0, the first stable version of its distributed inference stack.

  This open-source, Kubernetes-native layer turns vLLM or SGLang model servers into one coordinated pool.

Extractive summary: sentences quoted from the sources.