RoboGen

A generative simulation framework where a large model proposes tasks, builds the simulated scene, and learns the skill.

RoboGen is a robot-learning framework proposed in November 2023 by researchers at CMU, MIT, the Tsinghua-affiliated IIIS, UMass Amherst, and other institutions, published at ICML 2024. Rather than having a large model output actions directly, it runs a “propose–generate–learn” loop: a large model first proposes a skill to learn, then automatically builds the matching simulated scene and objects, breaks the task into subtasks, and writes supervision signals such as reward functions; finally, depending on the task type, it picks reinforcement learning, motion planning, or trajectory optimization to learn the skill. This needs almost no manual task design, yet can continuously produce demonstration data for a wide range of skills, spanning articulated objects, deformable-object manipulation, and legged locomotion. The project page states it uses the Genesis simulation engine for physics and rendering. It's a representative example of “generative simulation.”

ExampleA large model proposes the task “pull out a suitcase's handle”; RoboGen automatically places a suitcase model into the scene, sets up the initial state, generates a reward function, and trains the skill with reinforcement learning.

Also called

RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation

Related

Generative Simulation · Genesis · Procedural Generation · LLM-based Task Planning · Reward Function · Simulation Data

Sources

RoboGen (arXiv:2311.01455)
RoboGen 项目主页 (Chinese)

As of

2024-06