OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents
We present OS-Helix, an end-to-end trained multimodal computer-using agent that unifies GUI actions and code execution in one action space.
ProofPaper ↗
Key points
- Real desktop automation is neither purely visual nor purely programmatic.
- In our preliminary experiments, naively adding a code tool degrades a GUI agent, revealing that GUI/code routing must be internalized through training on high-quality hybrid data.
- To construct the required supervision, we introduce application-artifact composition, pairing desktop applications with authentic file artifacts to instantiate executable Ubuntu states, and a decoupled instruction/evaluator generation pipeline that produces environment-grounded tasks whose final states can be verified regardless of the solution modality.
- For RL, we propose Step-balanced GSPO, which aggregates importance weight by action step rather than raw assistant tokens to prevent long code actions from dominating heterogeneous-action optimization.
Sources (2)
- [1]OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using AgentsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 05:14 PM
We present OS-Helix, an end-to-end trained multimodal computer-using agent that unifies GUI actions and code execution in one action space.
Real desktop automation is neither purely visual nor purely programmatic.
- [2]OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using AgentsHugging Face Daily Papers · Oct 15, 12:00 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026Active Generative Simulation: Meta-Learning What to Simulate for Sim-and-Real Policy Co-Training
- Oct 14, 2026ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models
- Oct 14, 2026VASPO: Self-Distillation with Privileged Value Estimation
- Oct 14, 2026Latent Compaction with Test-Time Training
- Oct 13, 2026Run fast, private agents on-device with LFM2.5-24B-A2B
- Oct 13, 2026unslothai/unsloth v0.1.907-beta: Large Performance Gains + More Browser Use