MMKG-R1: Constructing Multimodal Knowledge Graphs via Reinforcement Learning

Key points

• However, most existing methods construct MMKGs without explicit guidance on graph quality, while reinforcement learning (RL)-based methods that rely on final-answer correctness may provide only indirect feedback on MMKG quality.1
• To address these limitations, we propose MMKG-R1, a novel MMKG-quality-driven RL framework for training an MLLM as a multimodal knowledge constructor.1
• To be specific, MMKG-R1 introduces multiple quality-aware rewards to directly evaluate both the quality of information extracted from individual knowledge sources and the quality of the assembled MMKG.1
• Experiments across multiple datasets show that MMKG-R1 constructs MMKGs that improve downstream question-answering accuracy.1

Sources (1)

• [1]MMKG-R1: Constructing Multimodal Knowledge Graphs via Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 12:35 AM

  Multimodal knowledge graphs (MMKGs) organize information across different modalities to support multimodal Graph Retrieval-Augmented Generation (RAG), which aims to improve the performance of multimodal large language models (MLLMs).

  However, most existing methods construct MMKGs without explicit guidance on graph quality, while reinforcement learning (RL)-based methods that rely on final-answer correctness may provide only indirect feedback on MMKG quality.

Extractive summary: sentences quoted from the sources.