AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

A general vision-based teleoperation system that uses an ordinary camera to capture hand motion and drive many arms and dexterous hands.

AnyTeleop was proposed in July 2023 by Yuzhe Qin and colleagues at UC San Diego and NVIDIA, published at RSS 2023. The operator wears no glove or exoskeleton — they simply move in front of a camera: the system estimates hand and wrist pose from the image, then converts it through motion retargeting (mapping human hand joints onto a structurally different robot hand) into commands for a dexterous hand and arm, with collision avoidance built in. Earlier vision-based teleoperation systems were usually tied to one specific piece of hardware; AnyTeleop instead supports many arms, dexterous hands, and camera configurations within the same system, connecting to a real robot or to simulators such as SAPIEN and Isaac Gym, and also supporting remote browser viewing and multi-robot collaboration. The paper reports it performs better than systems custom-built for the same hardware. Its retargeting module has been open-sourced as the dex-retargeting library.

ExampleInside a simulator, an operator performs a grasping motion in front of a camera, and AnyTeleop drives a virtual arm and dexterous hand in real time to grasp the object, with the recorded trajectory usable directly for training an imitation-learning policy.

Related

Teleoperation · Motion Retargeting · dex-retargeting · Dexterous Manipulation · Open-TeleVision · Bunny-VisionPro

Sources

AnyTeleop (arXiv 2307.04577)
AnyTeleop 项目主页 (Chinese)

As of

2023-07