Ego-Exo4D

A large-scale video dataset led by Meta, capturing skilled human activities simultaneously from first-person and third-person viewpoints.

Ego-Exo4D is a multi-view video dataset released in 2023, led by Meta FAIR together with more than a dozen universities worldwide: about 1,286 hours of footage from 740 participants across 13 cities. During recording, participants wore Aria smart glasses to capture first-person (egocentric) video while 4–5 surrounding GoPro cameras captured third-person (exocentric) footage at the same time, covering 8 categories of skilled activity such as cooking, bike repair, dancing, and rock climbing, along with gaze data, IMU (inertial measurement unit) readings, 3D point clouds, camera poses, and expert narration transcripts. It's used to study what the same action looks like from the performer's own eyes versus from a bystander's viewpoint. For embodied AI, a robot's head- and wrist-mounted cameras see something close to a first-person view, while most instructional video on the internet is shot in third person, so this dataset can be used to learn the correspondence between the two viewpoints, as well as for hand and 3D human pose estimation.

ExampleThe same “bike repair” session is filmed simultaneously by Aria glasses and 4 GoPro cameras, letting researchers train a model to translate third-person footage into a first-person view — one of the cross-view translation benchmarks in the paper.

Related

Egocentric Video · Exocentric Video · Ego4D · Project Aria Glasses · Human Video Data · Hand Pose Estimation

Sources

Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (arXiv)
Ego-Exo4D 官网 (Chinese)

As of

2024-09