EgoMimic

A framework that co-trains a single policy on Project Aria human-hand data and robot data together.

EgoMimic is an imitation-learning framework proposed in 2024 by Danfei Xu's group at Georgia Tech. A human performs tasks while wearing Project Aria smart glasses, which record first-person video and 3D hand trajectories; this is paired with a low-cost bimanual robot deliberately designed to have kinematics close to human arms. During training, the human hand trajectories and robot actions are normalized into a shared representation, masks are applied over the human hands and the robot arms in the video to reduce visual appearance differences, and a single policy network is co-trained on both. In experiments, it outperforms imitation-learning methods trained on robot data alone on long-horizon, single-arm, and bimanual tasks, and generalizes to new scenes; the authors also found that adding 1 more hour of human hand data helps more than adding 1 more hour of robot data. It's a representative example of using cheap human first-person data to replace part of the teleoperation data needed; the same group later led the EgoVerse data platform.

ExampleOn a place-object-in-bowl task, EgoMimic trained on 2 hours of robot data plus 1 hour of human hand data clearly outperforms ACT trained on 3 hours of robot data alone.

Also called

EgoMimic: Scaling Imitation Learning via Egocentric Video

Related

Egocentric Video · Project Aria Glasses · Co-training · Human Video Data · Embodiment Gap · EgoVerse

Sources

EgoMimic: Scaling Imitation Learning via Egocentric Video (arXiv)
EgoMimic 项目主页 (Chinese)

As of

2024-10