VPT

OpenAI's approach of first training an inverse dynamics model to label online video with actions, then learning to play Minecraft by imitation.

VPT was released by OpenAI in June 2022, published at NeurIPS 2022. The difficulty with decision-making tasks is that while there's plenty of video online, none of it comes labeled with 'which key was pressed at each moment.' VPT first pays human contractors to play Minecraft while recording their keyboard and mouse input (about 1,962 hours), and uses this to train an inverse dynamics model (IDM), which infers the action taken between two frames; the IDM is then used to automatically pseudo-label about 70,000 hours of online gameplay video with actions, and behavior cloning trains a base policy on this data, which operates the game directly at 20 Hz through the same keyboard-and-mouse interface a human would use. This base policy already shows some zero-shot ability, and after further imitation learning and reinforcement-learning fine-tuning, it became the first system to synthesize a diamond tool in Minecraft from scratch (a task that takes a skilled human player about 20 minutes and 24,000 actions) — something reinforcement learning starting from nothing could essentially never achieve. The 'train an IDM on a little labeled data, then pseudo-label a lot of video' idea has since been widely borrowed in robotics.

ExampleAfter reinforcement-learning fine-tuning, the VPT base model can start from an empty inventory in Minecraft and work all the way through gathering materials and crafting tools to finally make a diamond tool, something essentially impossible for reinforcement learning trained from scratch.

Also called

Video PreTraining, Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos

Related

Inverse Dynamics Model · Pseudo Action Labels · Action-free Video · Behavior Cloning · Minecraft Environments · Latent Action Pretraining

Sources

Video PreTraining (VPT) (arXiv 2206.11795)
VPT 论文 HTML 版（数据规模细节） (Chinese)

As of

2022-06