SLEM: Spectrum-Aware Latent Expert Merging

Key points

• Mixture-of-Experts (MoE) models scale model capacity efficiently by activating only a small subset of experts for each input, while still requiring all parameters to be stored in memory, resulting in substantial memory overhead.1
• Expert merging techniques have recently emerged as a promising compression paradigm that reduces this overhead by consolidating multiple experts into a smaller set.1
• However, existing approaches have two key limitations: (i) direct convex interpolation in parameter space is not generally guaranteed to yield meaningful interpolation of expert functions; and (ii) estimating expert similarity or merging coefficients often relies on calibration data, introducing dependence on the calibration distribution.1
• SLEM groups experts within and across layers based on latent similarities and derives merging coefficients from expert weight spectra.1

Sources (1)

• [1]SLEM: Spectrum-Aware Latent Expert Merging
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 09:38 AM

  To address these limitations, we propose Spectrum-Aware Latent Expert Merging (SLEM), a calibration-free expert merging framework that occurs in a structured latent representation of expert weights using a VAE.

  Mixture-of-Experts (MoE) models scale model capacity efficiently by activating only a small subset of experts for each input, while still requiring all parameters to be stored in memory, resulting in substantial memory overhead.

Extractive summary: sentences quoted from the sources.