Language Models Learn to Improve Language Models

Key points

• However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges.1
• To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals.1
• To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL.1
• Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost.1

Sources (1)

• [1]Language Models Learn to Improve Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 11:51 AM

  Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement.

  However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges.

Extractive summary: sentences quoted from the sources.