Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift

Key points

• The Synergistic Policy and Reward Co-Evolving Framework (SPARK), originally proposed for reasoning tasks, exploits this idea by recycling training rollouts and their evaluation results to train a model not only to generate solutions but also to evaluate and improve them.1
• We investigate whether this policy-reward co-evolution approach transfers effectively to code generation and whether execution-grounded reflection provides an additional source of self-improvement.1
• We compare SPARK-style rollout recycling with execution-only Group Relative Policy Optimization (GRPO) using Qwen2.5-Coder-3B-Instruct.1
• We characterize these patterns as single-adapter policy drift: the shared trainable parameters increasingly depart from pretrained code-generation behavior without corresponding gains in generalization.1

Sources (1)

• [1]Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 01:48 PM

  Reinforcement learning with verifiable rewards (RLVR) is particularly well suited to code generation because generated programs can be objectively evaluated through execution.

  The Synergistic Policy and Reward Co-Evolving Framework (SPARK), originally proposed for reasoning tasks, exploits this idea by recycling training rollouts and their evaluation results to train a model not only to generate solutions but also to evaluate and improve them.

Extractive summary: sentences quoted from the sources.