DextrAH-G

NVIDIA's arm-and-hand dexterous grasping system, trained entirely in simulation, grasping and carrying objects continuously from depth images alone.

DextrAH-G is a dexterous-grasping method released by NVIDIA together with researchers at Stanford, the University of Utah, and Berkeley in July 2024, published at CoRL 2024. It controls a KUKA arm plus an Allegro dexterous hand — 23 motors in total — and the challenge is the high action dimensionality, the sim-to-real gap, and the need to avoid collisions and respect joint limits. The approach has three steps: train a teacher policy in Isaac Gym with reinforcement learning that can see privileged information such as the object's true pose; distill it into a student policy that sees only a depth image; then deploy zero-shot to the real robot. The key detail is that the policy's output doesn't go straight to the motors — it's handed to Geometric Fabrics, NVIDIA's reactive motion-generation method, which handles obstacle avoidance, joint limits, and posture, both protecting the hardware and making reinforcement-learning exploration easier.

ExampleIn a bin-picking test, DextrAH-G, using only a single RealSense depth camera mounted at the edge of the table, continuously grasped and carried objects with an 87% success rate over 256 attempts, at about 10.7 seconds per pick-and-place cycle.

Also called

DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabrics

Related

Geometric Fabrics · Dexterous Manipulation · Teacher-Student Distillation · Privileged Information · Sim-to-Real Transfer · Isaac Gym

Sources

DextrAH-G (arXiv:2407.02274)
DextrAH-G 项目主页 (Chinese)

As of

2024-10