Bunny-VisionPro

A teleoperation system that uses Apple Vision Pro to control two dexterous hands in real time, with vibration feedback, for imitation-learning data collection.

Bunny-VisionPro is an open-source teleoperation system released in July 2024 by researchers at the University of Hong Kong and Xiaolong Wang's lab at UC San Diego, built for collecting bimanual dexterous-manipulation demonstrations for imitation learning. It uses an Apple Vision Pro headset to track the operator's hands and wrists in real time and maps that motion onto two xArm-7 robot arms fitted with two 6-DoF Ability dexterous hands (24 degrees of freedom in total). Finger retargeting uses an optimization method that aligns human and robot fingertip keypoints. The system includes built-in avoidance of collisions and singular configurations (poses where the arm loses the ability to move in some direction), and uses low-cost vibration motors to feed the robot hand's fingertip touch signals back to the operator. The paper used data collected with the system to train ACT, Diffusion Policy, and DP3, reaching higher success rates than data collected with baseline teleoperation systems.

ExampleAn operator wearing a Vision Pro headset remotely controls both arms and hands to complete long, multi-step tasks like wiping a window, sweeping the floor, or making coffee; when the robot's fingertips touch something, small vibration motors give the operator real-time touch feedback.

Also called

Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning

Related

VR Teleoperation · Apple Vision Pro · AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System · Bimanual Manipulation · Dexterous Manipulation · Motion Retargeting

Sources

Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning (arXiv 2407.03162)
Bunny-VisionPro 项目主页 (Chinese)

As of

2024-07