Scaling Parameter-Efficient Fine-Tuning for Brain Encoding

Key points

• Brain encoding models aim to predict neural responses to stimuli.1
• Trained with a low-rank adapter (LoRA) and a lightweight readout, our approach consistently improves brain encoding performance across nine diverse multimodal backbones.1
• We also show the adapted multimodal backbones remain interpretable: modality attribution recovers the canonical sensory topography that prior work often obtains by construction from separate unimodal encoders, and neural adaptation increases the contribution of audio inputs and cross-modal mixing within the backbone.1
• To scale out the number of adapted models, we develop a multi-LoRA kernel that accelerates joint training of multiple LoRAs on a shared backbone.1

Sources (2)

• [1]Scaling Parameter-Efficient Fine-Tuning for Brain Encoding
  Hugging Face Daily Papers · Oct 19, 12:00 AM

  We show that parameter-efficient fine-tuning (PEFT) enables us to effectively adapt very large pretrained backbones for brain encoding with a small number of trainable parameters.

  Brain encoding models aim to predict neural responses to stimuli.

• [2]Scaling Parameter-Efficient Fine-Tuning for Brain Encoding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:46 PM · same content

Extractive summary: sentences quoted from the sources.