Joint, Decoupled, or Agent: When Each Multimodal Alignment Paradigm Wins
Jointly pretrained multimodal encoders are a common way to combine modalities, while alternatives integrate their information during downstream training or inference.
ProofPaper ↗
Key points
- We compare three paradigms, instantiated with public checkpoints, across 16 paired datasets in five domains and multiple training-label budgets.
- DECOUPLED trains a fusion head on frozen, independently pretrained encoders.
- ScienceQA, MELD, and MetMeme shift from AGENT to fusion as supervision increases.
- We test whether representation and task diagnostics predict these preferences on held-out datasets before fitting their candidate systems.
Sources (1)
- [1]Joint, Decoupled, or Agent: When Each Multimodal Alignment Paradigm WinsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 06:35 PM
Jointly pretrained multimodal encoders are a common way to combine modalities, while alternatives integrate their information during downstream training or inference.
We compare three paradigms, instantiated with public checkpoints, across 16 paired datasets in five domains and multiple training-label budgets.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026Deep LLM layers expose passage-level context in EEG during silent reading
- Oct 15, 2026AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents
- Oct 15, 2026SAFETYEDIT: Sparse Internal Targets with Prompt-Wide Control for LLM Jailbreak Backdoors
- Oct 15, 2026RAG-RST: Retrieved Exemplars as In-Context Discourse Schema for RST Parsing
- Oct 15, 2026Close Enough is Nowhere Near: Fidelity-Consumability Decoupling in KV Caches
- Oct 15, 2026Incentivizing Tool-Use Reinforcement Learning for Post-Arrival Emergency Care Prediction