Real-to-Sim-to-Real

Recreating a real scene in simulation, training or generating data there, and then deploying the resulting policy back to the real robot.

Real-to-sim-to-real chains real-to-sim and sim-to-real into a single pipeline: a real environment is scanned and rebuilt as a digital twin inside a simulator; reinforcement learning, large-scale synthetic-data generation, or safe evaluation is then carried out in that recreated scene; and the resulting policy is finally transferred back to the real robot. Compared with plain sim-to-real, the training scene is a direct copy of the deployment scene, so the sim-to-real gap is smaller; compared with using only real-robot data, simulation allows cheap trial and error and deliberately introduced perturbations, which improves robustness. A common recent approach reconstructs photorealistic imagery with 3D Gaussian Splatting and pairs it with a physics engine to handle interaction. The limitation is that every new scene requires reconstructing it again from scratch, and physical parameters such as friction and softness remain hard to identify accurately.

ExampleMIT's RialTo builds a scene's digital twin in about 25 minutes using scanning tools like Polycam, brings roughly 15 real demonstrations into simulation to fine-tune with reinforcement learning, and then distills the result back into a vision-based real-robot policy; the paper reports over 67% higher robustness than pure imitation learning across 8 tasks such as stacking plates and placing books on a shelf.

Also called

Real2Sim2Real

Related

Real-to-Sim · Sim-to-Real Transfer · Digital Twin · Gaussian Splatting-based Simulation · Reinforcement Fine-Tuning (RL Fine-Tuning) · Sim-to-Real Gap (Reality Gap)

Sources

Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation (RialTo, arXiv 2403.03949)
RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator (arXiv 2411.11839)