World Models

A classic 2018 world-model paper that trained an agent's policy entirely inside a 'dream' the model learned on its own.

World Models is a paper released in March 2018 by David Ha (Google Brain) and Jürgen Schmidhuber (NNAISENSE), published the same year at NeurIPS under the title 'Recurrent World Models Facilitate Policy Evolution.' The agent has three parts: V, a variational autoencoder that compresses a 64×64 image into a vector of a few dozen dimensions; M, a mixture density network combined with a recurrent neural network that predicts the probability distribution of the next vector; and C, a linear controller with only about a thousand parameters, trained with the evolutionary strategy CMA-ES. The key result is that the controller can be trained entirely inside a 'dream' generated by M, and still work when placed back into the real VizDoom game. This paper made the term 'world model' popular again, and later work such as the Dreamer series continues down this path.

ExampleOn the CarRacing-v0 driving task, the agent scored 906±21, ahead of prior methods' 591–838; on the VizDoom fireball-dodging task, a controller trained only inside the dream scored 1092±556 once transferred back to the real game.

Also called

Recurrent World Models Facilitate Policy Evolution, World Models (Ha & Schmidhuber, 2018)

Related

World Model · Learning in Imagination · Variational Autoencoder · Recurrent Neural Network · Mixture Density Network · DreamerV3

Sources

World Models (arXiv:1803.10122)
World Models 交互式论文页 (Chinese)

As of

2018-12