Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning

Key points

• Language models are often evaluated alone but increasingly deployed together, routed between, made to debate, ensembled, or collaboratively trained.1
• We call this a model's compositional strength and formalize it using the Shapley value of a cooperative game over coalitions of a model pool, and use a model's leave-one-out contribution to the full coalition as a tractable proxy, requiring n+1 rather than 2^n system evaluations.1
• We then propose compositional reinforcement learning (C-RL), which trains a policy inside a live multi-model system and rewards the outcome produced by the system as a whole, rather than the policy's correctness in isolation.1
• Across three collaboration protocols and eight tasks spanning reasoning, coding, and scientific discovery, C-RL produces the strongest collaborator in 21 of 24 settings, lifting collaborative performance by +4.0 to +5.4 pts on average ( 3x the gain of baselines), up to +13.1 pts.1

Sources (1)

• [1]Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 10:16 PM

  Evaluating individual vs. compositional strengths across five model pools, twelve collaboration methods, and twelve datasets, we find them largely uncorrelated: in 12 of 15 (pool, method-family) settings, individual strength does not translate into compositional advantages, and the coupling is weakest among general-purpose models that practitioners are most likely to deploy.

  Language models are often evaluated alone but increasingly deployed together, routed between, made to debate, ensembled, or collaboratively trained.

Extractive summary: sentences quoted from the sources.