Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen LLM Hear Beyond the Transcript
We propose DuplexJev, which feeds ASR-encoder hidden states through a small connector into a frozen LLM and reads each question as a single-token distribution over its options.
ProofPaper ↗
Key points
- Full-duplex voice agents make many small, closed decisions, which current systems answer by slow autoregressive decoding.
- DuplexJev also hears the speaker: gender and emotion accuracy both reach 90% (from 55% and 28%) with a cross-attention connector, whose spoken QA drops by only 1 point (83% to 82%).
- We train decisions with cross-entropy on the read-out answer token, instead of the usual transcript distillation, whose teacher never hears the voice, and keep distillation for content.
- Encoders and LLMs are interchangeable; we release weights, training recipe, a batched-inference pipeline for full-duplex serving and a bilingual spoken-QA set.
Sources (1)
- [1]Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen LLM Hear Beyond the TranscriptHugging Face Daily Papers · Oct 2, 12:00 AM
We propose DuplexJev, which feeds ASR-encoder hidden states through a small connector into a frozen LLM and reads each question as a single-token distribution over its options.
Full-duplex voice agents make many small, closed decisions, which current systems answer by slow autoregressive decoding.
Extractive summary: sentences quoted from the sources.
Before this
- Sep 30, 2026axolotl-ai-cloud/axolotl v0.20.0
- Sep 29, 2026[AINews] AMD buys World Labs for $8.2B, as Atlas solves sparse reconstruction problem for robotics, design and more
- Sep 29, 2026Salt++: Context-Aligned Post-Training for Few-Step Streaming Multimodal Generation
- Sep 28, 2026Notes on NVIDIA Nemotron
- Aug 22, 2026sgl-project/sglang v0.5.18
- Aug 10, 2026huggingface/transformers v5.15.0: Release: v5.15.0