Sim-to-Real Transfer

Training a robot policy in simulation, then deploying it to work on a real robot.

Sim-to-real transfer means training a robot policy in a simulator first, then deploying it on a real robot. Simulation can run thousands of environments at once, and a failure costs nothing, so reinforcement-learning policies for quadruped and humanoid walking and dexterous manipulation are mostly trained in simulation first. The difficulty is the sim-to-real gap: friction, mass, motor response, and visuals in simulation never fully match reality, and a policy can fail once it reaches a real robot. Common countermeasures include domain randomization (randomizing physical and visual parameters during training so the real world is just one more case the policy has seen), system identification (calibrating simulation parameters from real-robot data), actuator modeling (using real-robot data to model the simulated motors so they respond like the real ones), and teacher-student distillation (first training a teacher policy that can see the simulator's full state, then having a student policy that only uses sensors available on the real robot imitate it). In 2017, Tobin and colleagues trained an object detector using only simulated images with randomized textures and achieved about 1.5 cm of localization error in the real world — an early landmark result.

ExampleETH Zurich's ANYmal quadruped: a walking policy is trained with reinforcement learning in simulation and combined with an actuator network learned from real-robot data, then deployed directly to the real robot, where it runs faster than earlier methods and can even get back up after falling.

Also called

Sim2Real

Related

Sim-to-Real Gap (Reality Gap) · Domain Randomization · System Identification · Actuator Modeling (Actuator Network) · Real-to-Sim · Teacher-Student Distillation

Sources

Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World (Tobin et al., 2017)
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (Zhao et al., 2020)
Learning agile and dynamic motor skills for legged robots (Hwangbo et al., 2019)