Handheld Gripper Data Collection

A human holds a camera-equipped gripper and performs tasks directly, recording footage and gripper motion as robot training data.

Handheld gripper data collection is a way to gather data without a real robot present: a collector holds a gripper matching the exact model used on the robot's end effector, with a camera mounted on it, and performs the task directly in the real world; afterward, algorithms recover the gripper's pose trajectory and opening width from the video to serve as action labels. The best-known example is UMI, from Stanford, Columbia, and the Toyota Research Institute at RSS 2024: a wrist-mounted GoPro with a 155° fisheye lens and side mirrors uses visual-inertial SLAM (simultaneous localization and mapping) to track the trajectory. It is cheaper and more portable than teleoperation, letting data collection leave the lab entirely — UMI reports collection speeds more than 3 times faster than teleoperation. The precondition is that the gripper and camera viewpoint must match between collection and deployment.

ExampleUMI uses a handheld gripper to collect demonstrations for dishwashing, two-arm cloth folding, and throwing objects into bins; the resulting policy deploys directly onto a robot arm and can complete tasks in environments and with objects it never saw during collection.

Also called

UMI-style Collection

Related

Universal Manipulation Interface · FastUMI · AgileX Pika · Robot-free (Embodiment-free) Data Collection · Capture-Execution Isomorphism · DexUMI

Sources

Universal Manipulation Interface (UMI) 项目主页 (Chinese)
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots (arXiv 2402.10329)