EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

Key points

• Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation.1
• Beyond model scaling, this architecture has demonstrated the potential to decouple factual knowledge storage from general-purpose computation, offering a promising route to updating factual knowledge while keeping the Transformer backbone fixed.1
• Realizing this potential is challenging because different expressions of a fact may activate different n-gram embeddings, while updating shared embeddings can unintentionally change the model's predictions about other facts.1
• EngramEdit first computes target memory representations that make the model predict the updated fact across multiple expressions.1

Sources (1)

• [1]EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 05:58 PM

  We propose EngramEdit for decoupled knowledge updates through conditional memory.

  Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation.

Extractive summary: sentences quoted from the sources.