OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Key points

• We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory.1
• The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations.1
• Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query.1
• With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.1

Sources (2)

• [1]OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory
  Hugging Face Daily Papers · Oct 19, 12:00 AM

  Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains.

  We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory.

• [2]OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:59 PM · same content

Extractive summary: sentences quoted from the sources.