Generative Simulation

Using foundation models to automatically generate simulation tasks, scenes, and training supervision, producing robot training data at scale.

Generative simulation is an approach proposed by Zhou Xian and colleagues in a 2023 paper: rather than having a large model output robot actions directly, use foundation models — language models, image and 3D generative models — to fully automatically generate diverse tasks, scenes, and training supervision (such as reward functions and demonstration trajectories), and learn skills at scale inside simulation. It targets the problem that manually building scenes and designing tasks and rewards is slow and limits data diversity. A typical pipeline: a large model first proposes a task, then retrieves or generates 3D assets to build the scene, breaks the task into sub-steps, and uses reinforcement learning or motion planning to generate data and train a policy. Notable projects include RoboGen, GenSim, and Holodeck. It differs from procedural generation (building randomized scenes from hand-written rules) in that it is driven by foundation models rather than fixed rules.

ExampleRoboGen's loop works like this: a large model proposes a task and skill to learn, selects relevant object assets and arranges them into a plausible simulated scene, breaks the task into sub-tasks, then chooses its own learning method, generates training supervision, and trains a policy — all without manual task design.

Also called

Automatic Simulation-Scene Generation, Automatic Simulation-Task Generation

Related

Procedural Generation · Synthetic Data · Simulation Data · RoboGen · Holodeck · Genesis

Sources

Towards Generalist Robots: A Promising Paradigm via Generative Simulation (arXiv 2305.10455)
RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation (arXiv 2311.01455)