DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation

Key points

• Our intuition is that understanding what an action will change starts with understanding where things are: complementary RGB and depth observations ground a common scene representation from which to infer those changes.1
• Guided by this intuition, we introduce DepState-JEPA, a dual-stream RGB--Depth joint-embedding predictive architecture that connects spatial understanding with action-conditioned prediction through a shared latent state, DepState.1
• Robot-state readouts provide recursive proprioceptive inputs; object-trajectory supervision and calibrated forward kinematics shape state learning through motion targets and joint--TCP consistency.1
• Comparisons show stronger RGB-encoder transfer after RGB-D adaptation, feature completion, conditioned visual forecasting, and manipulation success.1

Sources (1)

• [1]DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 05:51 AM

  Predictive representations for robot manipulation should connect scene geometry with the state changes induced by actions.

  Our intuition is that understanding what an action will change starts with understanding where things are: complementary RGB and depth observations ground a common scene representation from which to infer those changes.

Extractive summary: sentences quoted from the sources.