Mixture-of-Memories: Low-Overhead Language Modeling with Modular Knowledge Injection

Key points

• Large language models store substantial factual knowledge in their parameters, yet this knowledge is difficult to control, update, and reliably elicit at inference time.1
• MoM combines a frozen base LM with two neural memory experts: a latent memory that provides low-latency hidden-state corrections and a lexical memory that provides a token-space knowledge pathway.1
• A lightweight Budget Router sets per-token mixture weights from uncertainty and disagreement features, replacing the fixed interpolation of prior memory modules.1
• Experiments across language modeling, factual QA, update analysis, and efficiency show that MoM improves both predictive accuracy and updateability while retaining low inference overhead.1

Sources (1)

• [1]Mixture-of-Memories: Low-Overhead Language Modeling with Modular Knowledge Injection
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 05:32 PM

  We propose Mixture-of-Memories (MoM), a memory-augmented language modeling framework that separates routed neural memory from writable factual memory.

  Large language models store substantial factual knowledge in their parameters, yet this knowledge is difficult to control, update, and reliably elicit at inference time.

Extractive summary: sentences quoted from the sources.