Dobb-E

An open-source household-robot framework from NYU and Meta, released in 2023, that teaches a robot a new task from just 5 minutes of demonstration.

Dobb-E is an open-source household robot-learning framework released in November 2023 by Lerrel Pinto's group at NYU together with Meta, in a paper titled On Bringing Robots Home. Real homes vary enormously in lighting, furniture, and objects, so policies trained in a lab often fail once moved into a house, while collecting robot data inside people's homes is expensive. The team built a demonstration tool called the Stick: a $25 reacher-grabber fitted with a few 3D-printed parts and an iPhone, which an ordinary person can hold and use to record demonstrations just by going about a chore. Using it, they recorded 13 hours of data across 22 New York households (the HoNY dataset), pretrained a visual representation called HPR (built on ResNet-34) with self-supervised learning, and deployed it on the commercial mobile robot Hello Robot Stretch, where a new task needs only a handful of demonstrations plus brief fine-tuning. Code, data, models, and hardware designs are all open-source. Dobb-E is an early example of the “handheld gripper data collection” approach, and the same team later built Robot Utility Models.

ExampleOver about 30 days of testing across 10 households in the New York area, Dobb-E attempted 109 household tasks, using just 5 minutes of demonstration and 15 minutes of model adaptation per new task, reaching an overall success rate of 81%.

Also called

On Bringing Robots Home, An Open-Source, General Framework for Learning Household Robotic Manipulation

Related

Handheld Gripper Data Collection · Universal Manipulation Interface · Hello Robot Stretch · Household Tasks · Pre-trained Visual Representation · Robot Utility Models

Sources

On Bringing Robots Home (arXiv 2311.16098)
Dobb·E 项目主页 (Chinese)

As of

2023-11