MIPS: Mitigating Mode Collapse in Reinforcement Learning for Large Outcome Spaces

Key points

• Inverse Probability Scaling (IPS) counters this in group-based policy gradients such as GRPO by dividing each reward by the outcome's probability, estimated from its frequency within the sampled group.1
• We propose Marginal IPS (MIPS), which performs marginal evaluation by combining the sampled trajectory with a learned model of the trajectories that lead to each outcome.1
• On Hypergrids, where the exact sampling distribution is computable, MIPS stays near the reward-distribution target as the outcome space grows relative to the rollout group, while frequency-based IPS degrades.1
• In molecular synthesis and phylogenetic inference, MIPS sustains diverse, high-quality sampling where GRPO and frequency-based IPS collapse, showing that MIPS serves as a reliable solution to mode collapse in RL.1

Sources (1)

• [1]MIPS: Mitigating Mode Collapse in Reinforcement Learning for Large Outcome Spaces
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 02:42 AM

  Expected-reward maximization in reinforcement learning concentrates probability mass on a few outcomes, even in tasks that require many distinct, high-quality solutions, a failure known as mode collapse.

  Inverse Probability Scaling (IPS) counters this in group-based policy gradients such as GRPO by dividing each reward by the outcome's probability, estimated from its frequency within the sampled group.

Extractive summary: sentences quoted from the sources.