Toward a Universal Local Speech Feature Extractor Through Distillation
Self-supervised speech models commonly consist of a set of convolutional neural network (CNN) layers used as a local feature extractor, followed by transformer layers.
ProofPaper ↗
Key points
- The CNN layers are computationally demanding, and are also similar across models and resemble the much simpler mel spectral features.
- We hypothesize that the CNN layers can be replaced by a simple, universal model.
- To test this hypothesis, we learn a two-layer feature extractor via distillation from one or more teacher models (HuBERT, data2vec, and WavLM).
- Experiments on SUPERB(-SG) benchmark tasks show that our model largely retains the performance of the teacher models and can also be used in an unseen model (wav2vec 2.0).
Sources (1)
- [1]Toward a Universal Local Speech Feature Extractor Through DistillationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 03:40 AM
Self-supervised speech models commonly consist of a set of convolutional neural network (CNN) layers used as a local feature extractor, followed by transformer layers.
The CNN layers are computationally demanding, and are also similar across models and resemble the much simpler mel spectral features.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026Local proxies for memory-efficient training
- Oct 16, 2026Agentic In-Context Learning: What Algorithms Can Transformers Learn in Closed-Loop Interaction?
- Oct 15, 2026huggingface/transformers v5.20.0: Release v5.20.0
- Oct 15, 2026huggingface/trl v1.16.0
- Oct 15, 2026Physics-Aware Distillation of a Grid Foundation Model for Real-Time Edge Inference
- Oct 15, 2026FocusOPD: Structure-Guided On-Policy Distillation for Diffusion Models