VAttnPO: Visual Attention-Guided Policy Optimization
We introduce VAttnPO (Visual Attention-Guided Policy Optimization), which uses answer-conditioned visual attention to refine token-level credit while preserving the verifier-derived trajectory-mean advantage.
ProofPaper ↗
Key points
- Multimodal reinforcement learning with verifiable rewards commonly assigns one sequence advantage to an entire response.
- This outcome-level signal distinguishes overall response quality but leaves changes in visual-evidence use across generated tokens implicit.
- VAttnPO measures their agreement in visual-attention space using Jensen--Shannon divergence, rather than reading guidance from teacher output-token distributions or introducing an attention-matching objective.
- Trajectory centering, relative-position weighting, and credit balancing turn this agreement into a zero-mean, position-aware correction to the GRPO sequence advantage.
Sources (1)
- [1]VAttnPO: Visual Attention-Guided Policy OptimizationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 01:04 PM
We introduce VAttnPO (Visual Attention-Guided Policy Optimization), which uses answer-conditioned visual attention to refine token-level credit while preserving the verifier-derived trajectory-mean advantage.
Multimodal reinforcement learning with verifiable rewards commonly assigns one sequence advantage to an entire response.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation
- Oct 12, 2026Look Back to Move Forward: Learning to Revisit for Agentic Compositional Image Generation
- Oct 11, 2026RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
- Oct 11, 2026Apertus 1.5 Release Notes
- Oct 10, 2026vllm-sr/d3
- Oct 10, 2026KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization