JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs

Key points

• LLM agents are evolving from a single model calling tools to planners that orchestrate multiple expert models whose capabilities live in distinct weights.1
• This shift breaks a hidden assumption of today's serving systems: that the working set of models fits in GPU memory.1
• On the memory-constrained GPUs typical of private and edge deployments, naive whole-model swapping serializes weight transfer with computation and, without host-backed KV restoration, discards session state at a switch.1
• JointServe exploits three structural properties of agent workloads (directed calls, gapped execution, and dual-state residency) to prepare models before they are called: an intent signal parsed from the caller's own output stream triggers prefetch during generation, partial k residency reduces switching to incremental layers, and a preemptive transfer scheduler protects live-session KV with a guaranteed bandwidth share.1

Sources (1)

• [1]JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 08:59 AM

  We present JointServe, a serving system that treats weight shards and KV blocks as residency objects competing for one unified memory budget.

  LLM agents are evolving from a single model calling tools to planners that orchestrate multiple expert models whose capabilities live in distinct weights.

Extractive summary: sentences quoted from the sources.