GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations

Key points

• Through this paper, we attempt to solve this problem by building GlobeBench, a novel benchmark with 1,632 tasks across five separate domains and nineteen simulation skills.1
• The tasks check whether models can predict the environment's response to a specific action given the full agent trajectory by understanding earlier state changes, following environment rules, and predicting the effects of actions on both related and unrelated objects.1
• We compare the predicted environment response with an execution-derived reference, using a hybrid architecture with both deterministic gates and a semantic judge.1
• We find that model accuracy decreases as actions affect a larger portion of the environment.1

Sources (2)

• [1]GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 10:24 PM

  Text-based world models have recently shown potential to simulate complex RL environment state transitions for a diverse set of use cases.

  Through this paper, we attempt to solve this problem by building GlobeBench, a novel benchmark with 1,632 tasks across five separate domains and nineteen simulation skills.

• [2]GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations
  Hugging Face Daily Papers · Oct 15, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.