RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

Key points

• Neural speech codecs provide discrete tokens for speech generation.1
• Variable-frame-rate codecs address this trade-off by adapting token allocation to speech content, but threshold-based allocation lets content determine the token count, so the sequence length a speech generator must produce varies across utterances and datasets at a fixed setting.1
• A model trained on large-scale multilingual speech achieves competitive reconstruction quality across English, Japanese, and Mandarin, with the best intelligibility preservation among the compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates.1
• These results demonstrate that budgeted adaptive speech representations support both high-quality reconstruction and streaming text-to-speech generation.1

Sources (1)

• [1]RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding
  Hugging Face Daily Papers · Oct 21, 12:00 AM

  We present RubatoCodec, a variable-frame-rate speech codec built on X-Codec-2 with an exact utterance-level event budget and streaming decoding.

  Neural speech codecs provide discrete tokens for speech generation.

Extractive summary: sentences quoted from the sources.