DIFFLET: A Fast Inference Engine for Diffusion Transformer Models on AWS Trainium

Key points

• On AWS Trainium2, every component of a diffusion model runs as a program compiled ahead of time for fixed shapes, while pipelines differ across models, prompt lengths across requests, and caching decisions across denoising steps.1
• A small probe program computes the caching signal on the device and returns one number per step, and a resident model process keeps every program loaded across requests.1
• Across five image and video model families, fused attention accelerates denoiser steps by 1.2-4.3x over the same framework using PyTorch/XLA attention, and at 32.8k visual tokens it is the only attention that compiles.1
• On four of the five models, resident serving is 1.5-3.4x faster than standalone runs that reload weights from a warm file cache for each generation.1

Sources (1)

• [1]DIFFLET: A Fast Inference Engine for Diffusion Transformer Models on AWS Trainium
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 07:25 AM

  We present Difflet, a serving engine in which five model pipelines share one compilation backend, joint attention is split across cores by attention head, and a host controller decides at each denoising step whether to run the denoiser or reuse its earlier output.

  On AWS Trainium2, every component of a diffusion model runs as a program compiled ahead of time for fixed shapes, while pipelines differ across models, prompt lengths across requests, and caching decisions across denoising steps.

Extractive summary: sentences quoted from the sources.