ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence
We present ConvexLLM, a general-purpose speech and audio language model whose C-Gate interface represents audio as continuous convex combinations of frozen text embeddings.
ProofPaper ↗
Key points
- Speech and audio language models connect continuous acoustic representations to decoders pretrained primarily on text data.
- Most such models use MLP or Q-Former interfaces to map acoustic features into the decoder's input space, requiring the decoder to learn how to interpret the resulting representations.
- We investigate whether grounding these representations in the decoder's existing text embeddings can help it use acoustic information while better preserving its pretrained text capabilities.
- In a comparison with a 2B-parameter decoder, C-Gate achieves higher aggregate spoken-language performance than MLP and Q-Former interfaces, with text-task performance close to MLP and task-dependent tradeoffs in transcription and audio understanding.
Sources (1)
- [1]ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio IntelligencearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 02:35 PM
We present ConvexLLM, a general-purpose speech and audio language model whose C-Gate interface represents audio as continuous convex combinations of frozen text embeddings.
Speech and audio language models connect continuous acoustic representations to decoders pretrained primarily on text data.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026ChatGPT can peek at your Chrome tabs now
- Oct 15, 2026Anthropic gives outside AI watchdog “employee-like” access
- Oct 15, 2026TIPS: A Tiny Inference-time Policy for Scaling in Diffusion Language Models
- Oct 15, 2026NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning
- Oct 15, 2026Anthropic tells investors quarterly revenue up nearly 50% to $16.8bn ahead of IPO
- Oct 15, 2026Welcome FLUX.3 Image Klein in Diffusers