DivAlgo: A systematic evaluation of algorithmic diversity for code generation models

Key points

• Code generation models are actively being used in professional and educational settings.1
• However, the competence of large language models (LLMs) to generate algorithmically diverse code remains largely unknown.1
• We also introduce a lightweight similarity metric to enable large-scale evaluation.1
• Our experiments across 24 modern code generation models found that uncontrolled model-generated code is significantly less diverse than human solutions (capturing 31% of human diversity), and even the best diverse code generation strategy could achieve only 70% of human diversity.1

Sources (1)

• [1]DivAlgo: A systematic evaluation of algorithmic diversity for code generation models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 08:24 PM

  We propose the DivAlgo benchmark, a large-scale dataset constituting algorithmically diverse solutions to programming problems.

  Code generation models are actively being used in professional and educational settings.

Extractive summary: sentences quoted from the sources.