Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning

Key points

• However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information.1
• Contexts of equal length can pose very different challenges: a task may be solvable from a single local passage, or it may require integrating evidence distributed across the full context.1
• We construct LURCH, a multi-domain question-answering benchmark of 6,550 instances with supporting evidence and reasoning graphs, enabling all three properties to be quantified.1
• These findings establish evidence scope, dispersion and reasoning depth as measures of long-context reasoning difficulty that complement context length.1

Sources (1)

• [1]Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 03:28 PM

  Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty.

  However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information.

Extractive summary: sentences quoted from the sources.