Neural Simulator

A simulator learned from data with a neural network, predicting what happens next instead of relying on hand-written physics equations.

A neural simulator is a simulator that learns from data, rather than from hand-written physics equations, how a scene changes given the current state and an action. One line of work learns physical quantities directly, such as DeepMind's Graph Network Simulator (GNS, ICML 2020), which represents fluids, rigid bodies, and deformable materials as particles and predicts their motion. A second line learns directly on pixels, which is essentially an action-conditioned video world model: UniSim (2023) learned an interactive simulator from internet and robot data, and policies trained inside it transferred zero-shot to a real robot; in September 2024, 1X trained a world model on thousands of hours of EVE robot data to evaluate policies. Neural simulators can absorb real-world complexity that is hard to hand-model, such as cloth, but objects can deform or vanish inconsistently, physics isn't guaranteed to be conserved, and errors accumulate over long rollouts. They are commonly used for policy evaluation, data generation, and learning in imagination.

Example1X's world model takes a robot's starting frame plus a sequence of actions and generates the resulting video, which is then used to compare different policy versions on how likely each is to complete tasks like folding laundry or opening a door.

Also called

Learned Simulator, Neural Simulation, World-Model Simulator

Related

World Model · UniSim · 1X World Model · World-Model-based Policy Evaluation · Physics Engine · Learning in Imagination

Sources

Learning to Simulate Complex Physics with Graph Networks (arXiv 2002.09405)
Learning Interactive Real-World Simulators (UniSim, arXiv 2310.06114)
1X World Model