Cross-Modal Latent Influence: Patch-Level Training Sample Attribution for Vision-Language Models
Recent latent-mediated influence methods provide feature-level attribution in language models, but lack the multimodal and spatial structure needed to ground influence in VLMs. We introduce MOSAIC (Multimodal Orthogonalized Sparse Autoencoders for Influence Characterization).
ProofPaper ↗
Key points
- Explaining a vision--language model (VLM) prediction requires tracing training-example influence to the multimodal evidence that mediates it, yet existing attribution methods largely operate only at the sample level.
- It learns a shared sparse image--text latent space whose concepts are 1) sparse within each modality, 2) supported on contiguous image regions, 3) aligned to the phrases that describe them, and 4) approximately decorrelated.
- We evaluate MOSAIC on Qwen and Gemma across captioning, chest X-ray understanding, and visual question answering.
- We find that visual evidence can substantially reorganize training-data attribution while preserving more of the shared latent structure.
Sources (1)
- [1]Cross-Modal Latent Influence: Patch-Level Training Sample Attribution for Vision-Language ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 06:04 PM
Recent latent-mediated influence methods provide feature-level attribution in language models, but lack the multimodal and spatial structure needed to ground influence in VLMs. We introduce MOSAIC (Multimodal Orthogonalized Sparse Autoencoders for Influence Characterization).
Explaining a vision--language model (VLM) prediction requires tracing training-example influence to the multimodal evidence that mediates it, yet existing attribution methods largely operate only at the sample level.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026Task Context Shapes Component Roles in Language Models
- Oct 9, 2026X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation
- Oct 9, 2026Taking Tokenization off the Host for Agentic LLM Serving
- Oct 9, 2026Attention Sees, MLP Refuses: Tracing the Attention-to-MLP Safety Pathway in LLMs
- Oct 9, 2026Language Models Have Interpretable Narrative Mechanisms
- Oct 8, 2026BeliefScope: Diagnosing Evidence-Driven Revision and Pressure-Induced Shifts in Large Language Models