Manifold-Constrained Residual Adaptation with Frame-Adaptive Layer Aggregation for Speaker Verification

Key points

• Building on our preliminary study of manifold-constrained Hyper-Connections in speaker encoders, we explore parameter-efficient adaptation of both residual paths and feature selection across layers.1
• We first propose a Manifold-Constrained Residual Adapter (MCRA), which splits the hidden dimensions of each Transformer residual state into N groups and uses a small Sinkhorn-normalized matrix to exchange information among them.1
• We further propose Frame-Adaptive Layer Aggregation (FALA), an attention-based aggregation method that uses a single learned query shared across all utterances to compute layer weights separately for each speech frame.1
• Compared with a frozen WavLM baseline using a fixed layer-weighted sum, our method achieves relative EER reductions of 22.4% on VoxCeleb1-O and 2.9% on CN-Celeb, with trainable adaptation parameters equivalent to only 0.0009% and 0.0016% of the WavLM-Large parameters, respectively.1

Sources (1)

• [1]Manifold-Constrained Residual Adaptation with Frame-Adaptive Layer Aggregation for Speaker Verification
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 06:47 AM

  Large self-supervised speech models provide powerful representations for speaker verification, but full fine-tuning is expensive.

  Building on our preliminary study of manifold-constrained Hyper-Connections in speaker encoders, we explore parameter-efficient adaptation of both residual paths and feature selection across layers.

Extractive summary: sentences quoted from the sources.