Cache2LoRA: Understanding and Improving Context Internalization

Key points

• Controlled diagnosis of existing context-to-LoRA methods separates fact discrimination, target retention, and response adaptation.1
• We introduce Cache2LoRA, which uses each query to aggregate evidence from a reusable frozen document cache before generating LoRA updates.1
• Across 17 tasks, Cache2LoRA improves on existing context-to-LoRA methods and achieves macro F1 of 0.663, approaching full-context in-context learning at 0.671.1
• At matched memory size, generated LoRA preserves answer accuracy more reliably than the learned-KV variant under long, diverse intervening text and combines more effectively with retained document KV, yielding higher hybrid F1 and EM.1

Sources (1)

• [1]Cache2LoRA: Understanding and Improving Context Internalization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 06:35 PM

  Context-to-LoRA methods encode documents as low-rank parameter updates, allowing language models to answer without retaining the source text in their decoding context.

  Controlled diagnosis of existing context-to-LoRA methods separates fact discrimination, target retention, and response adaptation.

Extractive summary: sentences quoted from the sources.