MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision

Key points

• Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal.1
• This leaves open whether multimodal large language models (MLLMs) can integrate time series with real-world visual evidence.1
• Evaluating eight open-source and proprietary MLLMs reveals a shared bottleneck: representing time series as text creates very long token sequences and weakens fine-grained alignment with visual evidence, especially in long-video settings.1
• We then introduce TimeOmni-v, a multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head.1

Sources (1)

• [1]MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 07:20 AM

  We introduce MMTA (multimodal temporal analysis), a benchmark of tri-modal tasks with diverse cross-modal complementarity patterns among time series, text, and vision.

  Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal.

Extractive summary: sentences quoted from the sources.