Segment Where You Look: Attention as the Interface Between Language Models and Segmenters

Key points

• The dominant interface compresses it into the hidden state of a single [SEG] token, a d-dimensional vector with no spatial index, and leaves the decoder, which never sees the expression, to recover the referent's extent.1
• We show that the VLM already computes a better signal: the [SEG] token's attention over the image tokens, which is defined on the patch grid by construction.1
• Probing a frozen segmentation VLM, we find that this attention yields 18 to 22 more gIoU than the [SEG] hidden state when read by the same probe, that it gains nothing from the image where the hidden state does, and that it is concentrated in the late layers.1
• Building on these findings, we introduce AttnSeg, which replaces the [SEG] embedding with this attention as the decoder's prompt.1

Sources (1)

• [1]Segment Where You Look: Attention as the Interface Between Language Models and Segmenters
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:06 PM

  Vision-language models (VLMs) that segment from language must pass the location of the referent to a pixel decoder.

  The dominant interface compresses it into the hidden state of a single [SEG] token, a d-dimensional vector with no spatial index, and leaves the decoder, which never sees the expression, to recover the referent's extent.

Extractive summary: sentences quoted from the sources.