ResearchResearch paperRobotics & Embodied AI1 source · Oct 20, 2026

COWBOY: A Scalable Sim-to-Real Framework for Learning Contextual Whole-Body Manipulation

To address this, we present COWBOY, a scalable sim-to-real framework for learning such contextual whole-body manipulation.

Mobile manipulators must adapt not only to objects, but also to the spatial context in which they appear: where to position the base, how to manipulate, and where to look while navigating surrounding scene geometry.
On whole-body dexterous grasping, COWBOY transfers zero-shot to hundreds of objects and diverse in-the-wild scenes, spanning cluttered tabletops, shelves, drawers, and other unstructured settings.
Rather than learning scene adaptation directly with RL in simulation, a vectorized environment-aware whole-body controller (WBC) handles reaching in thousands of scenes by coordinating base-arm motion, collision avoidance, and active vision.
The resulting policy exhibits contextual whole-body behavior, automatically adapting base placement, camera viewpoint, arm trajectory, and grasp strategy to observed scene and object geometry.

[1]COWBOY: A Scalable Sim-to-Real Framework for Learning Contextual Whole-Body Manipulation
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 09:59 PM
To address this, we present COWBOY, a scalable sim-to-real framework for learning such contextual whole-body manipulation.
Mobile manipulators must adapt not only to objects, but also to the spatial context in which they appear: where to position the base, how to manipulate, and where to look while navigating surrounding scene geometry.

Extractive summary: sentences quoted from the sources.

Before this