ReproBAIT: Can coding agents build their own biological AI tools?

Key points

• Large language models are increasingly competent at leveraging scientific knowledge and existing computational tools to execute complex workflows.1
• We evaluated a set of biological AI tools of varying complexities: ProteinMPNN, an inverse protein folding tool, RFdiffusion3, an all-atom generative diffusion tool, and a closed-source molecular design tool.1
• Across a range of leading closed- and open-source coding agents, we find substantial reproduction capability for some selected tools, though limited capability for a highly complex tool.1
• We find moderate evidence consistent with memorisation, varying across coding agents.1

Sources (1)

• [1]ReproBAIT: Can coding agents build their own biological AI tools?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 08:50 AM

  We introduce an evaluation titled ReproBAIT (Reproducing Biological AI Tools), which measures the ability of coding agents to design, implement, and train biological AI tools taken from published research across a range of information access conditions.

  Large language models are increasingly competent at leveraging scientific knowledge and existing computational tools to execute complex workflows.

Extractive summary: sentences quoted from the sources.