Conjugate Compilation: Self-Improving Agentic Co-Compilation for Speculative Decoding

Key points

• As LLM adoption surges, scaling model serving requires unlocking stranded GPU capacity alongside deploying more GPUs.1
• Our key insight is that cross-model compilation should be informed by the measured performance of the co-located shared execution it produces: an optimization that improves a kernel in isolation need not improve the models when they execute together.1
• We realize CONJUGATE COMPILATION through a self-improving agentic search system in which an LLM edits cross-model optimization plans using the compiler and GPU execution as its environment.1
• Across six single-GPU speculative-decoding configurations on NVIDIA B300 and RTX 5090 GPUs, CONJUGATE COMPILATION achieves a geometric-mean per-round speedup of 1.37x over conventional execution.1

Sources (1)

• [1]Conjugate Compilation: Self-Improving Agentic Co-Compilation for Speculative Decoding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 04:46 AM

  We introduce CONJUGATE COMPILATION, which reframes speculative decoding as a cross-model compilation problem and jointly optimizes the draft and target models within their shared execution rather than treating each in isolation.

  As LLM adoption surges, scaling model serving requires unlocking stranded GPU capacity alongside deploying more GPUs.

Extractive summary: sentences quoted from the sources.