LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition

Key points

• Using singular value decomposition, we find the minimal subspace that keeps 99% of the factor energy.1
• A 48-LoRA sub-pool of LoRA-Hub on FLAN-T5 tasks shows comparable overlap (62%), while the diversity-curated LoRA-Retriever benchmark sits near the no-overlap baseline (96%), so pool composition rather than modality predicts where compression is large.1
• We build on this overlap with LoRASpace, a training-free method that computes the subspace once per pool and reuses it in two inference modes, a static mode at merge cost and a dynamic mode for input-dependent routing.1
• On the diffusion pools, static LoRASpace matches composite quality at merge cost, and dynamic LoRASpace is more than 6x faster than composite with six active LoRAs on a 17-LoRA pool, mainly because it runs one backbone pass instead of six.1

Sources (1)

• [1]LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 12:46 AM

  Low-rank adaptation (LoRA) attaches small low-rank matrices to a frozen backbone, and public repositories host many LoRAs per backbone, each for a different concept or task.

  Using singular value decomposition, we find the minimal subspace that keeps 99% of the factor energy.

Extractive summary: sentences quoted from the sources.