Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image Captioning
To address this, we propose a reinforcement learning framework to mitigate object hallucination in LVLM-based image captioning, driven entirely by natural language processing (NLP) metrics.
ProofPaper ↗
Key points
- Large Vision-Language Models (LVLMs) have excelled in joint visual and language understanding, particularly in image captioning.
- While optimizing reinforcement learning solely on NLP metrics offers scalability, it also introduces pitfalls: reward hacking, sparse feedback, and costly training.
- Our framework sidesteps these by introducing an Object Consistency Score (OCS) capturing both correctness and coverage of objects, incorporating KL-regularization for denser token-level guidance, and designing a lightweight PPO variant that fine-tunes adapters atop a frozen LVLM backbone, achieving efficiency and stability.
- These results reveal a broader insight in multimodal learning: LVLMs can effectively self-correct hallucinations using metric feedback, motivating deeper exploration into scalable reinforcement frameworks that significantly reduce human effort.
Sources (1)
- [1]Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image CaptioningarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 02:47 PM
To address this, we propose a reinforcement learning framework to mitigate object hallucination in LVLM-based image captioning, driven entirely by natural language processing (NLP) metrics.
Large Vision-Language Models (LVLMs) have excelled in joint visual and language understanding, particularly in image captioning.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 10, 2026CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance
- Oct 10, 2026From Video to Vision: Learning Generalizable Humanoid Carrying from Human Motion
- Oct 9, 2026Visualizing the World Model in an Agent's Value Function
- Oct 9, 2026Goal-Oriented Bundle Management and Flow Control for Deep-Space Communications via Reinforcement Learning
- Oct 9, 2026The Thinking Leakage in Reinforcement Learning for LLMs
- Oct 9, 2026Rethinking Relative Policy Differences Transfer of On-Policy Distillation