Tactile Simulation

Simulating what a tactile sensor would read inside a physics simulator, so tactile-equipped policies can be trained in sim.

Tactile simulation means generating the output a tactile sensor should produce inside a physics simulator — for instance, the tactile image from a visuotactile sensor (a sensor with a built-in camera that photographs the deformation of an elastic gel pad, such as GelSight or DIGIT), the displacement of markers embedded in the gel, or the pressure map from a piezoresistive array. The hard part is handling contact, soft-material deformation, lighting, and image formation all at once — the result has to look real and still be fast to compute. Common approaches fall into three groups: rendering an image directly from contact geometry, as in Meta's open-source TACTO; calibrating a lookup table from a small number of real samples, as in Carnegie Mellon's Taxim; and computing contact force fields and images in parallel on the GPU, as in NVIDIA's TacSL. It lets sim-to-real transfer extend to contact-rich tasks like peg insertion and screwing, though a gap between simulated and real sensor readings remains, usually addressed with domain randomization or a small amount of real-robot data.

ExampleThe TACTO paper used simulation to generate tactile data for 1 million grasps, training a model to predict whether a given grasp would be stable.

Also called

Tactile Sensor Simulation

Related

Vision-Based Tactile Sensor · TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors · Taxim: An Example-based Simulation Model for GelSight Tactile Sensors · TacSL: A Library for Visuotactile Sensor Simulation and Learning · Sensor Simulation · Tactile Image

Sources

TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors (arXiv 2012.08456)
Taxim: An Example-based Simulation Model for GelSight Tactile Sensors (arXiv 2109.04027)
TacSL: A Library for Visuotactile Sensor Simulation and Learning (arXiv 2408.06506)