Learning How to Remember: Memory Modality Selection for GUI Agents
Recent advances in vision-language models (VLMs) have substantially improved the capabilities of GUI agents, while memory augmentation has emerged as a promising approach for reusing knowledge from past interactions.
ProofPaper ↗
Key points
- It remains unclear whether a single memory modality is consistently optimal, or how modality preference changes across interaction steps.
- In this work, we conduct a systematic study of memory modality in VLM based GUI agents.
- Across GUI benchmarks, we find that the preferred memory representation is strongly step dependent.
- Motivated by the finding, we introduce SwitchMem, a routing framework that dynamically selects between textual and visual memory representations at each step.
Sources (2)
- [1]Learning How to Remember: Memory Modality Selection for GUI AgentsHugging Face Daily Papers · Oct 13, 12:00 AM
Recent advances in vision-language models (VLMs) have substantially improved the capabilities of GUI agents, while memory augmentation has emerged as a promising approach for reusing knowledge from past interactions.
It remains unclear whether a single memory modality is consistently optimal, or how modality preference changes across interaction steps.
- [2]Learning How to Remember: Memory Modality Selection for GUI AgentsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 08:15 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026Redesigning source control at Anthropic
- Oct 12, 2026Scheduled Agents, Dreaming and Q&A: Claude Managed Agents (Part 3 of 4) — Anthropic
- Oct 12, 2026AReA: Adaptive Receptive-Field Attention for Efficient High-Resolution Diffusion Transformers
- Oct 12, 2026AUGUstus: Autonomous Driving VLA Model with Unified Generation and Understanding
- Oct 12, 2026Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards
- Oct 11, 2026openai/openai-python v3.29.0