Human Motion Generation

A model that generates a sequence of 3D human skeletal poses from a condition such as a text description.

This kind of model takes a description (such as 'a person walks forward and then sits down') or a motion category as input and outputs a frame-by-frame sequence of 3D human skeletal poses. A common training set is HumanML3D, released in 2022, with 14,616 motion clips and 44,970 text descriptions, with the motions drawn from motion-capture datasets such as AMASS. The representative work is MDM (Human Motion Diffusion Model), proposed by Tevet and colleagues and published at ICLR 2023: it implements a diffusion model with a Transformer, and at each step predicts the clean motion directly instead of the noise, which makes it easy to add geometric constraints like foot contact; the same model can also do motion in-betweening, interpolation, and editing of individual body parts. This kind of model was originally built for animation and games; in embodied AI, the generated human motion can be retargeted (mapping human joints onto robot joints) and handed to a humanoid's motion-tracking controller, making it one route to 'make a humanoid do something with a sentence.'

ExampleGiven MDM the prompt 'a person walks forward and then sits down,' it outputs a 3D skeletal animation of a few steps forward followed by sitting, which can then be retargeted onto a humanoid robot.

Also called

MDM

Related

MDM (Motion Diffusion Model) · HumanML3D · Diffusion Model · Motion Retargeting · Motion Tracking · Text-to-Motion

Sources

Human Motion Diffusion Model (Tevet et al., arXiv:2209.14916)
MDM 项目主页（ICLR 2023） (Chinese)
HumanML3D 数据集（GitHub） (Chinese)

As of

2023-05