KEEP-Bench: Evaluating Evidence Preservation in Context Compression

Key points

• Existing evaluations rely on QA metrics, but a reader can still answer correctly from an unfaithful compression, so these metrics underestimate such loss.1
• To bridge this gap, we introduce KEEP-Bench, a diagnostic benchmark that directly assesses compression faithfulness with 4,102 cases across three challenging settings that require keeping evidence for plausible reference resolutions, explicit qualifiers, and minority perspectives.1
• Experiments show that even with tailored prompting, the strongest models remain unreliable: despite context windows exceeding 100K tokens and inputs averaging only 3.1K tokens, the best open-weight model (31B) preserves all required evidence in only 66.8% of cases, and the best proprietary model in only 61.6%.1
• We further introduce a metric, Evidence Preservation Return (EPR), to relate evidence recall to inference FLOPs, and find that it peaks at moderate scales (4B-8B), where recall remains low.1

Sources (1)

• [1]KEEP-Bench: Evaluating Evidence Preservation in Context Compression
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:55 PM

  Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck.

  Existing evaluations rely on QA metrics, but a reader can still answer correctly from an unfaithful compression, so these metrics underestimate such loss.

Extractive summary: sentences quoted from the sources.