Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision

Key points

• Understanding contact and tactile magnitude in hand-object interaction is a key step toward physically intelligent systems, yet learning such representations is limited by the cost and restricted coverage of tactile measurements.1
• We first introduce a self-supervised pretraining objective that learns from video alone without tactile supervision.1
• Using video-derived hand geometry, Touch-JEPA predicts masked visual features at corresponding hand-surface locations from spatial and temporal context, encouraging the representation to capture latent contact dynamics.1
• With self-supervised pretraining alone, simple supervised readouts on frozen Touch-JEPA features outperform supervised baselines in contact discrimination and tactile magnitude estimation across three human-hand benchmarks, and transfer to robotic hands without robot tactile training data.1

Sources (1)

• [1]Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:56 PM

  We present Touch-JEPA, a framework for learning tactile representations from hand-object videos with limited tactile supervision.

  Understanding contact and tactile magnitude in hand-object interaction is a key step toward physically intelligent systems, yet learning such representations is limited by the cost and restricted coverage of tactile measurements.

Extractive summary: sentences quoted from the sources.