How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts

Key points

• We argue that the number of components a token needs is a detection question with a classical answer.1
• A token's responses to the rank-one components of a trained LoRA consist of a few large values on a Gaussian noise floor whose width matches that of an untrained adapter, and the floor carries cross-talk rather than adaptation.1
• We prove that the squelch is within a logarithmic factor of the oracle input-dependent allocation, that the selected threshold is risk-consistent, and that explaining away is a descent step toward the joint sparse code whose error is controlled by the coherence of co-active components.1
• On commonsense reasoning, knowledge, math, code, and GLUE with LLaMA-3.1-8B, Qwen3, and DeBERTaV3, SQUELCH improves over LoRA by 2.2--3.2 points and over the strongest mixture baseline by 0.8--1.2 points while activating 8.6 of 32 components per token on average.1

Sources (1)

• [1]How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 06:26 PM

  Low-rank adaptation (LoRA) applies every rank-one component of its update to every token, and mixture-of-experts variants replace this with a learned router that activates a fixed number of components.

  We argue that the number of components a token needs is a detection question with a classical answer.

Extractive summary: sentences quoted from the sources.