DexCap

Stanford's wearable hand motion-capture system that collects data by having a human use their own hands, for teaching robot dexterous hands.

DexCap is a portable hand-motion-capture data collection system from Fei-Fei Li's group at Stanford (first author Chen Wang), published at RSS 2024, paired with DexIL, an algorithm for learning policies from human hand data. Teleoperating a dexterous hand to collect data is slow, expensive, and requires a robot to be physically present; DexCap instead lets a person wear a device and perform tasks directly with their own hands. Rokoko electromagnetic motion-capture gloves measure each finger's position relative to the palm and aren't disrupted by objects blocking the view; a chest-mounted rig carries an RGB-D LiDAR camera plus three SLAM tracking cameras that record wrist pose and a point cloud of the scene; a mini PC and battery pack in a backpack support about 40 minutes of continuous collection. DexIL uses inverse kinematics to convert the human hand motion into motion for a LEAP robot hand, then trains a policy with point-cloud-based imitation learning; a human can also step in to correct the policy during deployment.

ExampleThe paper demonstrates policies trained with only about 30 minutes of human hand motion-capture data and no teleoperation at all, including two-handed tasks like making tea and cutting things with scissors.

Also called

DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

Related

Motion Capture · Data Glove · Robot-free (Embodiment-free) Data Collection · DexUMI · LEAP Hand · Human-in-the-Loop

Sources

DexCap 项目主页 (Chinese)
DexCap (arXiv)
DexCap GitHub 仓库（RSS 2024） (Chinese)

As of

2024-03