Open-TeleVision

An open-source system for teleoperating a humanoid robot by wearing a VR headset and seeing stereo video from the robot's own viewpoint.

Open-TeleVision is an open-source teleoperation system from Xiaolong Wang's group at UC San Diego and MIT (CoRL 2024). The operator wears an Apple Vision Pro headset (the code also supports Meta Quest 3); when they turn their head, the robot's pan-tilt unit or neck turns to match, and a head-mounted ZED Mini stereo camera streams stereo video back to the headset in real time, so the operator sees a 3D view from the robot's own viewpoint; arm and finger motion is mapped onto the robot through the dex-retargeting library. It targets the limitations of ordinary teleoperation — a fixed camera view, no sense of depth, and difficulty with fine manipulation. The authors collected demonstrations on a Unitree H1 and a Fourier GR-1, training policies with a modified version of ACT to perform tasks like sorting cans and folding clothes, and also demonstrated teleoperation across a distance of about 3,000 miles.

ExampleThe operator wears an Apple Vision Pro and turns their head toward the table; the Unitree H1's head unit turns to match. As the operator reaches out to grab something, the robot's dexterous hand sorts cans into different boxes in sync, and the whole process is recorded as imitation-learning demonstration data.

Also called

TeleVision, OpenTeleVision, OpenTV

Related

VR Teleoperation · Apple Vision Pro · dex-retargeting · Action Chunking with Transformers · Unitree H1 · PH2D

Sources

Open-TeleVision: Teleoperation with Immersive Active Visual Feedback (arXiv 2407.01512)
Open-TeleVision 项目主页 (Chinese)
OpenTeleVision/TeleVision (GitHub)

As of

2024-07