Real-to-Sim

Recreating a real scene, its objects, and the robot inside a simulator, so the simulation matches reality as closely as possible.

Real-to-sim is the reverse step of sim-to-real: information is captured from the real world and used to build a matching version inside a simulator. It involves geometry and appearance reconstruction (scanning a room and its objects with a phone, NeRF, or 3D Gaussian Splatting), identifying physical parameters (mass, friction, joint damping, and so on), and aligning the robot's controller; the resulting simulated version is often called a digital twin. It serves two main purposes. One is evaluation: a real test setup is recreated in simulation, and the simulated score is used to predict real-robot performance, saving a large amount of real-robot testing. The other is training: data is generated or reinforcement learning is run inside the recreated scene, and the result is transferred back to the real robot — this full loop is called real-to-sim-to-real. The difficulty is that any error in the reconstructed visuals or physics becomes a new sim-to-real gap of its own.

ExampleSIMPLER (SimplerEnv) builds matching simulated environments for real experiments with the Google Robot and the WidowX arm, focusing on narrowing both control and visual gaps; the paper shows that policy performance inside it correlates strongly with real-robot results and can even reproduce how sensitive a policy is to various distribution shifts.

Also called

Real2Sim, Real-to-Sim Reconstruction

Related

Real-to-Sim-to-Real · Digital Twin · Sim-to-Real Transfer · Sim-to-Real Gap (Reality Gap) · SimplerEnv · Gaussian Splatting-based Simulation

Sources

Evaluating Real-World Robot Manipulation Policies in Simulation (SIMPLER, arXiv 2405.05941)
RialTo: Real-to-Sim-to-Real Approach for Robust Manipulation (arXiv 2403.03949)