Language Corrections

A person telling a working robot how to fix what it's doing, in words, while it keeps working.

Language corrections let a person adjust a robot's behavior in real time using natural language while it performs a task — saying things like “a little to the left” or “don't grab the cup yet.” Compared with re-demonstrating the task through teleoperation, speaking a correction is far cheaper and usable by non-experts. Google's 2022 Interactive Language trained a real-time policy that could follow roughly 87,000 distinct language instructions, letting a person watch and talk an arm through arranging blocks into a smiley face. Shi and colleagues' 2024 YAY Robot uses a hierarchical structure: a high-level policy issues language instructions, and a low-level policy executes them. A person's spoken corrections can change behavior on the spot and are also logged to keep training the high-level policy, without needing extra teleoperation data.

ExampleA robot keeps missing the opening while packing a bag; a person says “a little to the left,” and it adjusts immediately. The correction is logged, so next time the high-level policy issues a similar instruction on its own.

Also called

Verbal Corrections, Real-time Language Feedback

Related

Language-conditioned Policy · Human-in-the-Loop · Hierarchical Architecture · Hi Robot · RT-H · Failure Recovery

Sources

Interactive Language: Talking to Robots in Real Time
Yell At Your Robot: Improving On-the-Fly from Language Corrections

As of

2024-03