Evaluating Context Compaction on the Questions That Come Later

Key points

• Language-model agents compact their context for the question at hand, but the compacted context must also serve questions that come later.1
• On MultiRC, keeping the sentences that gpt-6-luna (through the OpenAI Decisions API) judges necessary for the current question raises its accuracy on that question by 11.4 points at 25% of the tokens.1
• Deleting words instead of sentences does better: LLMLingua-2, which ignores the question, beats random sentence deletion at the same token count by 12.5 points on the later MultiRC questions.1
• The ranking also depends on the setting: on Evidence Inference sentence selection beats LLMLingua-2 at a nominal 25% budget, and on $τ$-bench trajectories, whose later tool calls need exact identifiers, it beats random deletion while LLMLingua-2 shows no detected gain.1

Sources (1)

• [1]Evaluating Context Compaction on the Questions That Come Later
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 09:15 PM

  We evaluate compaction on these later questions, using four corpora that annotate the evidence of several questions per text and recorded agent trajectories, and compare every method with random deletion at the same budget.

  Language-model agents compact their context for the question at hand, but the compacted context must also serve questions that come later.

Extractive summary: sentences quoted from the sources.