Omni-Modal LLMs Know More Than They Say
Omni-modal large language models (Omni-LLMs) extend LLMs to process heterogeneous modalities (e.g., vision, audio, and IMU signals) alongside language, enabling a broader range of multimodal tasks.
ProofPaper ↗
Key points
- Yet we find that these models often fail to express information from non-textual modalities even when they perceive it correctly.
- To investigate this phenomenon, we introduce Token-level Modality Alignment (TMA), which measures the alignment between individual modality representations and response representations.
- Motivated by these findings, we propose OmniCatalyst, a simple perceive $\rightarrow$ infer strategy that complements the original modality inputs with self-generated text, making non-textual knowledge more accessible.
- In summary, our findings uncover how Omni-LLMs can better express modality knowledge through language.
Sources (2)
- [1]Omni-Modal LLMs Know More Than They SayarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 03:52 AM
Omni-modal large language models (Omni-LLMs) extend LLMs to process heterogeneous modalities (e.g., vision, audio, and IMU signals) alongside language, enabling a broader range of multimodal tasks.
Yet we find that these models often fail to express information from non-textual modalities even when they perceive it correctly.
- [2]Omni-Modal LLMs Know More Than They SayHugging Face Daily Papers · Oct 12, 12:00 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 11, 2026TempoKV: Predictive KV Cache Quantization for Autoregressive Video Generation
- Oct 10, 2026Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models
- Oct 10, 2026Toward System One Clinical Intelligence: Parameter-Efficient Fine-Tuning and Counterfactual Explainability of Decision Foundation Models
- Oct 10, 2026SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability
- Oct 10, 2026LongHisDoc: A Comprehensive Benchmark for Chinese Long Historical Document Understanding
- Oct 9, 2026SAT-RL: Training Language Models for Logical Reasoning with Symbolic Solvers