Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents
We argue that a world model meant for verification should instead predict in the space in which observations are encoded, and present LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly, trained without semantic annotation on 1.85M real GUI transitions.
ProofPaper ↗
Key points
- A login screen that appears after a tap on Sign in is expected; the same screen after a tap on View order is an attack.
- For GUI agents, safety is therefore a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated by reusing a legitimate one.
- Existing GUI world models provide one, but they output it as text, code, or images, so checking it against the observed screen requires a second model to judge the two.
- We evaluate on RSWT-BENCH, a diagnostic where each credential screen appears under both a legitimate and a hijacked transition, so detectors that see only the screen are at chance by construction.
Sources (2)
- [1]Right Screen, Wrong Transition: World Models as Verifiers for GUI AgentsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 01:33 PM
We argue that a world model meant for verification should instead predict in the space in which observations are encoded, and present LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly, trained without semantic annotation on 1.85M real GUI transitions.
A login screen that appears after a tap on Sign in is expected; the same screen after a tap on View order is an attack.
- [2]Right Screen, Wrong Transition: World Models as Verifiers for GUI AgentsHugging Face Daily Papers · Oct 8, 12:00 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 7, 2026Sparse Planning in Visual World Models via Cost Gradients
- Oct 7, 2026UltraWorld: Learning Interactive Ultrasound World Models from Untracked Clinical Videos with Acoustic Sampling Map
- Oct 7, 2026RoboJEPA: Scaling Laws for Multi-Embodiment Robotic Latent World Models
- Oct 7, 2026DSReg: Provably Recovering Individual World Latents without Reconstruction
- Oct 6, 2026Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models
- Oct 6, 2026SPW-Nav Streams Language-Guided Panoramic Video in Real Time