Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing

Key points

• We present the Sequential Isolation Methodology, a controlled benchmarking and regression-testing protocol designed to reduce between-run measurement variance while deliberately varying workload concurrency.1
• We evaluate three representative open-source LLMs on an NVIDIA A100 80GB GPU using vLLM 0.9.1 across six context sizes and eight concurrency levels, with five repetitions per configuration.1
• The protocol is intended to provide a stable reference for reproducible comparison and regression testing rather than to predict absolute behavior under uncontrolled production traffic.1
• Infrastructure-as-Code and benchmark scripts support replication of the experimental environment.1

Sources (1)

• [1]Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 09:58 AM

  Reproducible benchmarking of Large Language Model (LLM) inference is challenging because repeated measurements can vary with execution and system state.

  We present the Sequential Isolation Methodology, a controlled benchmarking and regression-testing protocol designed to reduce between-run measurement variance while deliberately varying workload concurrency.

Extractive summary: sentences quoted from the sources.