Neural Trajectories

Synthetic robot training data generated by a video world model, with pseudo action labels added afterward.

Neural trajectories is a term used by NVIDIA's GEAR lab in GR00T N1 (March 2025) and DreamGen (May 2025) for synthetic robot data generated by a video world model. The pipeline has four steps: first, fine-tune a video generation model on the target robot's real data, so it learns that robot's appearance and how it moves; then, given a starting frame and a language instruction, generate video of the robot performing a task — which can involve entirely new motions or new environments never actually collected; next, use a latent-action model or an inverse-dynamics model (a model that infers the action from a pair of before-and-after frames) to add pseudo action labels to the generated video; finally, train a visuomotor policy on these pseudo-labeled videos together with real data. Compared with synthetic data from a physics simulator, this approach needs no simulated scenes or assets to be built, but the physical plausibility of the generated footage isn't guaranteed and needs dedicated evaluation — which is what DreamGen Bench was built for.

ExampleGR00T N1 generated about 827 hours of neural trajectories (roughly 10x) from about 88 hours of real-robot data, using 3,600 L40 GPUs over about 1.5 days; co-training with the real data raised the GR-1 humanoid's average success rate across 8 real-robot tasks by 5.8 percentage points.

Related

DreamGen · NVIDIA Isaac GR00T N1 · Synthetic Data · Pseudo Action Labels · Inverse Dynamics Model · Latent Action Model

Sources

DreamGen 项目主页（NVIDIA GEAR） (Chinese)
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots (arXiv 2503.14734)
DreamGen: Unlocking Generalization in Robot Learning through Video World Models (arXiv 2505.12705)

As of

2025-06