JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents
In this paper, we propose JAVIS, an end-to-end streaming framework for real-time audio-visual conversational agents.
ProofPaper ↗
Key points
- Human communication is inherently multimodal, conveying intent not only through linguistic and vocal cues but also through visual signals.
- On top of a shared LLM, a Joint Refined Head produces a unified audio-visual representation, which is fed into dedicated streaming branches for speech and face generation.
- In addition, we present JAVIS-Syn, a 334-hour synthetic corpus designed to address the lack of expressive audio-visual dialogue data, covering emotionally rich multi-turn conversations and assistant-style exchanges.
- Experiments show that JAVIS achieves approximately 200$\times$ lower initial latency than a strong baseline, while outperforming baselines in lip synchronization and facial expression appropriateness.
Sources (1)
- [1]JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational AgentsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 06:00 PM
In this paper, we propose JAVIS, an end-to-end streaming framework for real-time audio-visual conversational agents.
Human communication is inherently multimodal, conveying intent not only through linguistic and vocal cues but also through visual signals.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning
- Oct 15, 2026Diffusion Language Models as Open Quantum Systems
- Oct 15, 2026ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training
- Oct 15, 2026PriorityFEC: Deadline-Aware Streaming FEC for KV-Cache Handoff in Edge Multi-Agent Systems
- Oct 15, 2026Two Sample Sizes in Test-Time LLM Ensembling: Buffered ERM and Minimax Theory
- Oct 15, 2026CQDC-Warp: GPU-Accelerated Differentiable Quasi-Dynamic Simulation for Contact-Rich Manipulation