Which Long Documents Matter? A Controlled Study of Data Selection for Context Extension

Key points

• Random sampling of natural long documents (random-long) outperforms repacking at longer evaluation lengths.1
• Within the same candidate pool, selecting documents with high LongPPL key-token density yields further gains, unlike the evaluated generic-quality and LLM-judge selectors.1
• We introduce truncLongPPL to reduce scoring cost by computing key-token density over a truncated prefix.1
• After supervised fine-tuning (SFT), Qwen3-8B and OLMo 3-7B models trained with selected long documents outperform their random-long counterparts by 1.13--1.67 points on LongBench v2 and 0.62--1.44 points on MRCR.1

Sources (1)

• [1]Which Long Documents Matter? A Controlled Study of Data Selection for Context Extension
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 05:29 AM

  Natural long documents are widely used for context extension, but which selection pipelines are effective remains unclear.

  Random sampling of natural long documents (random-long) outperforms repacking at longer evaluation lengths.

Extractive summary: sentences quoted from the sources.