VirtualHome: Simulating Household Activities via Programs

A Unity-based simulation platform where virtual humans perform household chores driven by step-by-step 'programs.'

VirtualHome is a household-activity simulation platform proposed by MIT and the University of Toronto at CVPR 2018, built on the Unity3D game engine and controlled through a Python API. It writes chores like “making breakfast” or “watching TV” as “programs” — sequences of atomic actions such as walking to something, picking it up, opening it, and switching it on or off — which a virtual human then executes step by step inside a simulated apartment, rendering an annotated video as it goes. It is concerned with high-level task planning rather than simulating a robot arm's contact mechanics, so it is often used to test how well large language models decompose tasks. The current version, 2.3, supports procedural scene generation, and it has also spun off a human-robot collaboration benchmark called Watch-And-Help.

ExampleFor instance, “watch TV” can be written as the program “walk to the living room → pick up the remote → turn on the TV → sit on the sofa”; the virtual human executes it step by step, and the platform outputs the corresponding video and annotations.

Also called

VirtualHome

Related

Household Tasks · LLM-based Task Planning · Long-horizon Task · AI2-THOR · ALFRED · Simulator

Sources

VirtualHome: Simulating Household Activities via Programs (arXiv 1806.07011, CVPR 2018)
VirtualHome GitHub 仓库 (Chinese)

As of

2026-09