LEAPS: Long-context Evaluation of Adaptive Pattern Learning and State Tracking

Key points

• Long-context language models (LCLMs) power agents that act over long trajectories, from improving a codebase to running research experiments to forecasters that condition on lengthy histories.1
• These applications draw on three distinct long-context capabilities: recall, where the answer is a fact present in the context; state tracking, where the outcome can be computed by performing a series of transformations to a specified state; and pattern learning, where the goal is to make a prediction by sampling from a distribution inferred from many observations.1
• We evaluate 30 API and open models from 8K to 512K input tokens.1
• LEAPS distinguishes models that perform near perfectly on recall tasks, finding that such models' state tracking and pattern learning abilities can vary significantly and degrade much more dramatically with increasing input lengths--even for pattern learning, where more observations should make learning a theoretically easier task.1

Sources (1)

• [1]LEAPS: Long-context Evaluation of Adaptive Pattern Learning and State Tracking
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 07:19 AM

  In this work, we introduce LEAPS (Long-context Evaluation of Adaptive Pattern learning and State tracking), a collection of 12 novel datasets, spanning agentic workflows, behavioral modeling, and forecasting of market and world events.

  Long-context language models (LCLMs) power agents that act over long trajectories, from improving a codebase to running research experiments to forecasters that condition on lengthy histories.

Extractive summary: sentences quoted from the sources.