Gaussian Splatting-based Simulation

Rendering simulated scenes with 3D Gaussian Splatting reconstructions of real places, paired with a physics engine for motion.

This approach plugs 3D Gaussian Splatting (3DGS) — introduced by Kerbl and colleagues in 2023, which represents a scene as a large number of colored 3D Gaussian blobs and can render photorealistic images in real time — into robot simulation. Typically, a phone or camera scans a real scene to reconstruct it as 3DGS, and the objects and robot inside it are then aligned with the collision bodies used by the physics engine: the physics engine computes motion, while 3DGS handles rendering the camera view. The goal is to close the visual sim-to-real gap caused by traditional simulators' obviously synthetic-looking graphics, making it easier for policies trained on simulated vision to transfer zero-shot to a real robot, and enabling reproducible closed-loop evaluation. Notable projects include SplatSim, RoboGSim, and Real2Render2Real; AgiBot's Genie Sim 3.0 also added 3DGS scene reconstruction. Gaussian points carry no physical properties of their own, so collision geometry and a dynamics model still have to be supplied separately.

ExampleSplatSim replaced a simulator's usual mesh rendering with Gaussian-splatting rendering; an RGB manipulation policy trained only on simulated data then deployed zero-shot to a real robot at an average success rate of 86.25%, versus 97.5% for a policy trained on real-robot data.

Also called

3DGS Simulation, GS Simulation, Gaussian Splatting Simulator

Related

3D Gaussian Splatting · Real-to-Sim-to-Real · Sim-to-Real Gap (Reality Gap) · Digital Twin · RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator · Real2Render2Real

Sources

SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting (arXiv 2409.10161)
RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator (arXiv 2411.11839)
3D Gaussian Splatting for Real-Time Radiance Field Rendering (arXiv 2308.04079)

As of

2026-01