PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation

Rosy Chen¹, Mustafa Mukadam³ Michael Kaess² Tingfan Wu³ Francois R. Hogan³ Jitendra Malik³
Akash Sharma¹

¹Carnegie Mellon University, ²University of Washington, ³UC Berkeley, ⁴FAIR at Meta

PDF arXiv Video Code

PTLD: sim-to-real Privileged Tactile Latent Distillation is an approach to learn tactile dexterous policies without simulating tactile sensors. First, Privileged sensor policies are trained in simulation using reinforcement learning which produces strong policies. These policies are deployed in instrumented real-world setups to collect tactile demonstrations. Finally, a tactile state estimator is trained from tactile demonstrations to obtain robust real-world deployable tactile policies.

Abstract

Tactile dexterous manipulation is essential to automating complex household tasks, yet learning effective control policies remains a challenge. While recent work has relied on imitation learning, obtaining high quality demonstrations for multi-fingered hands via robot teleoperation or kinesthetic teaching is prohibitive. Alternatively, with reinforcement we can learn skills in simulation, but fast and realistic simulation of tactile observations is challenging. To bridge this gap, we introduce PTLD: sim-to-real Privileged Tactile Latent Distillation, a novel approach to learning tactile manipulation skills without requiring tactile simulation. Instead of simulating tactile sensors or relying purely on proprioceptive policies to transfer zero-shot sim-to-real, our key idea is to leverage privileged sensors in the real world to collect real-world tactile policy data. This data is then used to distill a robust state estimator that operates on tactile input. We demonstrate from our experiments that PTLD, can be used to improve proprioceptive manipulation policies trained in simulation significantly by incorporating tactile sensing. On the benchmark in-hand rotation task, PTLD achieves a 182% improvement over a proprioception only policy. We also show that PTLD enables learning the challenging task of tactile in-hand reorientation where we see a 57% improvement in number of goals reached over using proprioception alone.

Walkthrough Video

PTLD Simplified overview

Once the privileged sensor policies are trained in simulation, they are deployed in an instrumented real-world setup. In our specific instance (for both in-hand rotation and in-hand reorientation), we use 4 realsense cameras and ChArUco markers on the object to track the object pose at 50Hz. The robot is also equipped with tactile sensors -- Xela uSkin covering the fingertips, phalanges and the palm -- to collect tactile observations. Then the privileged sensor policies are rolled out in the real world to collect tactile data and privileged sensor latent pairs. This dataset is finally used to train a tactile state estimator, which we show recovers the privileged sensor latents quite well, and significantly outperforms deploying a proprioception policy.

In hand rotation

In-hand rotation policy trained purely with proprioception in simulation, deployed zero-shot in the real world. The policy results in the object falling out in approximately 13 seconds

In-hand rotation policy trained with PTLD, which incorporates tactile sensing, deployed in the real world. The policy rotates objects for over 2 minutes and shows recovery behaviors in gaits when slip occurs

PTLD robustness

In hand reorientation

In-hand reorientation is a challenging task that requires the robot to reorient an object in-hand without dropping it, and often requires complex finger gaits and recovery behaviors when slip occurs. We show that PTLD can be used to learn a tactile policy for this task, which significantly outperforms a proprioception-only policy.

Related research

Sparsh: Self-supervised touch representations for vision-based tactile sensing
Website
Self supervised perception for tactile skin covered dexterous hands
Website
Tactile beyond pixels: multisensory touch representations for robot manipulation hands
Website

BibTeX

If you find our work useful, please consider citing our paper:

@inproceedings{chen2026ptld,
  title = {PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation},
  author={Rosy Chen and Mustafa Mukadam and Michael Kaess and Tingfan Wu and Francois Hogan and Jitendra Malik and Akash Sharma},
  year = {2026},
  booktitle={arxiv},
  url={}
}