MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records

Key points

• Enterprise agents answer questions over workplace records that evolve through team collaboration.1
• Reliable question answering therefore requires agents to reason over sufficient evidence, resolve cross-source and temporal conflicts, and recognize when the available evidence is insufficient to establish an answer.1
• We evaluate 10 proprietary and open-source multimodal LLMs and find that even the strongest models struggle when evidence contains cross-source or temporal conflicts or spans multiple images.1
• These findings highlight the persistent challenges of reasoning over evolving workplace records and establish MMEnterprise-Bench as a testbed for developing more reliable enterprise agents.1

Sources (1)

• [1]MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 09:25 AM

  To systematically evaluate these capabilities, we introduce MMEnterprise-Bench, a benchmark comprising 1,613 questions grounded in evolving multimodal records from simulated collaboration projects, covering three categories: Evidence Reasoning, Conflict Resolution, and Unanswerability.

  Enterprise agents answer questions over workplace records that evolve through team collaboration.

Extractive summary: sentences quoted from the sources.