Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions

Key points

• Text language models (LMs) acquire broad linguistic and factual capabilities through large-scale pretraining, while comparable capabilities remain difficult to learn from speech alone.1
• In this work we introduce a speech-to-speech pretraining method that creates an internal text interface for knowledge transfer while preserving the speech-only input and output.1
• The model first consumes speech context without its transcription, switches to text prediction, and then uses the predicted text to lead future speech generation word by word.1
• Under matched training conditions, the proposed sequence construction substantially improves the generative perplexity of spoken continuations over speech-only and interleaved speech-text models, approaching the performance of a text-leading inner-monologue model with external ASR prompt transcription.1

Sources (1)

• [1]Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 10:20 PM

  Recent work has shown that knowledge distillation from pretrained text LMs offers a promising way to transfer their linguistic and factual capabilities to speech models.

  Text language models (LMs) acquire broad linguistic and factual capabilities through large-scale pretraining, while comparable capabilities remain difficult to learn from speech alone.

Extractive summary: sentences quoted from the sources.