ASSET: Adaptive Speech Segmentation via Reinforcement Learning
To this end, we introduce ASSET, a reinforcement learning (RL)-based framework that learns to adaptively compress encoder outputs by first partitioning them into variable-duration segments and then pooling each segment into a continuous audio token.
ProofPaper ↗
Key points
- Speech large language models (LLMs) are often built using pretrained acoustic encoders as an audio interface to an LLM decoder.
- We use group-relative policy optimization (GRPO) to train a segmenter with task performance feedback and a token-rate penalty.
- In automatic speech recognition (ASR) experiments, ASSET achieves performance comparable to a transcript-assisted character-boundary oracle with approximately 25% fewer audio tokens on English LibriSpeech, while improving on a phone-boundary oracle at a similar token rate on Mandarin AISHELL.
- Exploratory experiments on non-ASR tasks further illustrate how learned segmentation changes when the framework is adapted to tasks beyond transcription.
Sources (1)
- [1]ASSET: Adaptive Speech Segmentation via Reinforcement LearningarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 08:36 PM
To this end, we introduce ASSET, a reinforcement learning (RL)-based framework that learns to adaptively compress encoder outputs by first partitioning them into variable-duration segments and then pooling each segment into a continuous audio token.
Speech large language models (LLMs) are often built using pretrained acoustic encoders as an audio interface to an LLM decoder.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026Embedding Graph Kernel Entropy (EGKE): Fine-grained Hallucination Detection for LLMs from Embeddings
- Oct 9, 2026Collaborative Theory of Mind: Learning to Self-Revise by Anticipating User Responses
- Oct 9, 2026The Thinking Leakage in Reinforcement Learning for LLMs
- Oct 9, 2026Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding
- Oct 9, 2026SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models
- Oct 9, 2026ClawLoop: Self-Improving Agents via Harness-Native Training Loop