On the Limits of Scale: Inverse Scaling in Vision-Language Models

Key points

• Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements.1
• We evaluate 19 models from four distinct model families across eight multimodal benchmarks (nine evaluation splits), answering 29,386 questions per model, 558,334 model-question evaluations in total.1
• Using per-question pairwise analysis across all model pairs in each family, we identify 375 subtask-model-pair combinations exhibiting inverse scaling, spanning 4,342 unique queries where a smaller model answers correctly but a larger model fails, and we find that these patterns replicate across independently developed architectures.1
• We identify 13 subtask-dataset pairs that show inverse scaling across multiple families, consistent with the phenomenon being task-driven rather than model-specific.1

Sources (2)

• [1]On the Limits of Scale: Inverse Scaling in Vision-Language Models
  Hugging Face Daily Papers · Oct 19, 12:00 AM

  While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs) exhibit inverse scaling at the subtask level: specific question categories on which larger models perform worse than smaller counterparts within the same model family.

  Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements.

• [2]On the Limits of Scale: Inverse Scaling in Vision-Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 10:39 PM · same content

Extractive summary: sentences quoted from the sources.