Enhancing Multimodal RLVR with Pairwise Token Contrast and Adaptive Trajectory Confidence

Key points

• Recent self-distilled variants refine credit with a teacher--student likelihood gap.1
• To address this, we propose Pairwise token contrast and Adaptive trajectory ConfidencE (PACE).1
• Pairwise token contrast scores each response under four teacher views, crossing a correct or incorrect answer hint with the image present or removed, and compares the two hints for every token, which cancels shifts shared by both hints and reveals whether the token's support relies on the image.1
• Experiments show that both components matter: replacing pairwise token contrast with a single-view signal underperforms our method, while adaptive trajectory confidence adds further improvement.1

Sources (1)

• [1]Enhancing Multimodal RLVR with Pairwise Token Contrast and Adaptive Trajectory Confidence
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 10:55 AM

  Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but group-relative policy optimization broadcasts a single sequence-level advantage to every token of a response, although errors usually localize to a few decisive steps.

  Recent self-distilled variants refine credit with a teacher--student likelihood gap.

Extractive summary: sentences quoted from the sources.