Language-Table

Google's tabletop block-pushing dataset, with nearly 600,000 trajectories paired with natural-language instructions.

Language-Table is a dataset, simulation environment, and benchmark that the Google robotics team open-sourced alongside its 2022 paper Interactive Language: Talking to Robots in Real Time. The setup is simple: a UFACTORY xArm6 arm fitted with a cylindrical end effector pushes colored blocks around on a tabletop; the action space is just 2D planar displacement, at 5Hz. The data was collected using 4 robots and 10 teleoperators, and crowdworkers then reviewed the videos afterward, marking the start and end of each behavior and writing an instruction for it — that is, hindsight relabeling. In total there are nearly 600,000 language-labeled trajectories: 442,000 from the real robot and 181,000 from human-operated simulation, plus several script-generated simulation subsets. The resulting policy can execute roughly 87,000 different instruction phrasings, with an estimated success rate of 93.5%. It's mainly used to study real-time language interaction and instruction following, and was later folded into Open X-Embodiment.

ExampleAn operator can talk to the robot as they watch: first asking it to push one block next to another, seeing the result, then giving the next instruction, gradually building up a more complex long-horizon goal, with the robot responding to each instruction in real time.

Also called

Language Table, language_table

Related

Tabletop Manipulation · Language-conditioned Policy · Instruction Following · Hindsight Relabeling · Open X-Embodiment · PaLM-E

Sources

Interactive Language 项目主页 (Chinese)
google-research/language-table（GitHub）
Interactive Language: Talking to Robots in Real Time (arXiv 2210.06407)

As of

2022-10