Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs
In this work, we introduce HLR-AVSceneQA, a comprehensive benchmark with a hear-localize-reason hierarchy that evaluates whether models can recognize what is heard, localize sound sources in egocentric and allocentric views, and use these localized sources to reason about how they relate to the visual scene.
ProofPaper ↗
Key points
- Audio-visual large language models (AV-LLMs) have made strong progress in multimodal understanding, yet most treat audio as monaural semantic content, limiting their ability to reason about where sounds originate and how they relate to the visual scene.
- Recent spatial audio-visual studies address parts of this problem, but often focus on specific abilities, such as spatial correspondence or direction and distance reasoning.
- We further introduce HLR-LLM, which augments a strong audio-visual foundation model with a binaural spatial audio branch alongside its monaural semantic audio pathway.
- Experiments show that existing AV-LLMs remain limited in spatial grounding and relational reasoning, while HLR-LLM substantially improves spatial audio-visual scene understanding without compromising semantic audio-visual perception.
Sources (1)
- [1]Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 09:34 AM
In this work, we introduce HLR-AVSceneQA, a comprehensive benchmark with a hear-localize-reason hierarchy that evaluates whether models can recognize what is heard, localize sound sources in egocentric and allocentric views, and use these localized sources to reason about how they relate to the visual scene.
Audio-visual large language models (AV-LLMs) have made strong progress in multimodal understanding, yet most treat audio as monaural semantic content, limiting their ability to reason about where sounds originate and how they relate to the visual scene.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026Decomposing Brain-LLM Alignment with Interpretable Semantic Representations
- Oct 15, 2026Incentivizing Tool-Use Reinforcement Learning for Post-Arrival Emergency Care Prediction
- Oct 15, 2026Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference
- Oct 15, 2026Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries
- Oct 14, 2026Optimal Hydra: Model Heterogeneous Federated Learning for LLMs using Data-Informed Hydra-Nets
- Oct 13, 2026Plan or Retrieve First? A Scale Study of Tool Discovery Architectures