Neural Descriptor Fields

An MIT 2021 object representation that transfers a manipulation skill to new objects of the same category, in new poses, from just a few demonstrations.

Neural Descriptor Fields were proposed by Anthony Simeonov, Yilun Du, Pulkit Agrawal, Vincent Sitzmann, and colleagues at MIT, posted to arXiv in December 2021, published at ICRA 2022. It represents an object as a function: given any 3D point near the object, it outputs a descriptor vector, such that functionally equivalent locations on objects of the same category — the handle of different mugs, say — get similar descriptors. The descriptor comes from an occupancy network self-supervised on a 3D-reconstruction task, with no manual keypoint labeling needed. The network is SE(3)-equivariant (however the object rotates or moves, the descriptors transform the same way), so it can handle the object upright, on its side, or upside down. During a demonstration, the descriptors of a set of points near the gripper are recorded; at test time, optimization finds the gripper pose whose descriptors best match them.

ExampleGiven just 10 demonstrations of “grasp the mug's rim and hang it on a rack,” a Franka arm can hang up mugs it has never seen, in any orientation; the paper reports an overall success rate above 85%.

Also called

NDF, SE(3)-Equivariant Object Representations for Manipulation

Related

Object-centric Representation · Equivariant Policy / Equivariant Neural Network · Few-shot · Pick-and-Place · Pose · Imitation Learning

Sources

Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (arXiv 2112.05124)
NDF 项目页 (Chinese)

As of

2022-05