Holodeck

Given a one-sentence description, automatically generates an interactive 3D indoor scene using GPT-4 and Objaverse assets.

Holodeck is a collaboration between the University of Pennsylvania, Stanford, the University of Washington, and the Allen Institute for AI (Ai2), published at CVPR 2024. Embodied agents need large numbers of diverse training scenes, but building 3D rooms by hand is expensive. Holodeck lets a user type a description like “apartment for a researcher with a cat,” and GPT-4 supplies the common-sense knowledge: room layout, wall and floor materials, doors and windows, which objects belong in the scene, and spatial-relationship constraints between objects (such as a chair being next to a table); an optimization algorithm then solves for object placement, with objects retrieved from the Objaverse 3D asset library. The generated scene is loaded and used through AI2-THOR. In the paper's experiments, an object-goal navigation agent trained on Holodeck scenes generalized better zero-shot to new scene types such as music rooms and daycares.

ExampleGiven the input “apartment for a researcher with a cat,” Holodeck automatically plans the rooms, chooses wall and floor materials, picks furniture from Objaverse, and arranges it according to the constraints; the resulting scene can be loaded directly in AI2-THOR.

Also called

Language Guided Generation of 3D Embodied AI Environments

Related

AI2-THOR · ProcTHOR (Large-Scale Embodied AI Using Procedural Generation) · Objaverse · Generative Simulation · Procedural Generation · Object-Goal Navigation

Sources

Holodeck: Language Guided Generation of 3D Embodied AI Environments (arXiv 2312.09067)
allenai/Holodeck (GitHub)
Holodeck project page

As of

2024-06