When MLLMs miss what humans read: diagnosing and learning to read squint text

Key points

• Multimodal LLMs can fail to read text embedded in an image's coarse structure (text a human recovers by squinting) and report that no text is present.1
• We name this class Squint Text and study it as a recognition deficiency, motivated by cases from a production moderation pipeline.1
• Frozen-weight analysis shows that hidden text leaves a coarse signal while glyph identity is weakly represented; identity becomes substantially more accessible when the correct spatial regions are supplied, implicating spatial selection as a major bottleneck.1
• These findings motivate Squint OPSD: a frozen teacher receives a downscaled view and the ground-truth transcription as privileged context, and its token-level distributions supervise a student along on-policy trajectories.1

Sources (1)

• [1]When MLLMs miss what humans read: diagnosing and learning to read squint text
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 12:02 PM

  We introduce SquintBench, comprising 1076 production-derived moderation images, and SquintCtrl, a controlled synthetic suite for mechanism diagnosis.

  Multimodal LLMs can fail to read text embedded in an image's coarse structure (text a human recovers by squinting) and report that no text is present.

Extractive summary: sentences quoted from the sources.