Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding
To address these limitations, we propose Seg2Click, which reformulates GUI grounding as executable-region prediction: identifying where a click can validly execute the given instruction.
ProofPaper ↗
Key points
- Visual grounding is a central challenge for GUI agents, requiring an instruction to be mapped to a target within a crowded screenshot.
- Coordinate generation with single-point supervision reduces a spatially extended target to one position, despite the presence of many valid click locations.
- Using bounding-box supervision, Seg2Click predicts an executable-region mask and computes the final click point through a deterministic rule.
- Seg2Click-9B achieves 92.06%, 94.10%, and 55.22% on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro, respectively, while matched-backbone comparisons show improvements over both coordinate generation and patch-based grounding.
Sources (1)
- [1]Seg2Click: Learning Pixel-Level Action Masks for GUI Visual GroundingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 06:09 AM
To address these limitations, we propose Seg2Click, which reformulates GUI grounding as executable-region prediction: identifying where a click can validly execute the given instruction.
Visual grounding is a central challenge for GUI agents, requiring an instruction to be mapped to a target within a crowded screenshot.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026Interactive 4D Volumetric Liquid Forecasting under Moving-Solid Interaction
- Oct 12, 2026AReA: Adaptive Receptive-Field Attention for Efficient High-Resolution Diffusion Transformers
- Oct 11, 2026Rethinking Human Intent-to-CAD: Parametric CAD Model Generation via Cooperative Multi-Task Alignment and Spatial-Aware Reinforcement Learning
- Oct 11, 2026AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards
- Oct 10, 2026Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations
- Oct 10, 2026AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generation