Task-Oriented Grasping

Choosing a grasp based on what comes next — the same object gets held differently for different jobs.

Ordinary grasping cares only about holding an object securely; task-oriented grasping also requires the grasp to suit the task that follows — gripping a hammer by its handle to drive a nail, but holding a pair of scissors by the blade when passing it to someone else, leaving the handle for them. It has to connect object parts, affordances (which part of an object can be used for what), and task semantics, linking “picking something up” to “using it.” In 2020, Carnegie Mellon University and collaborators published the TaskGrasp dataset at CoRL (191 objects, 56 tasks, about 250,000 grasps), encoding object-task relationships with a knowledge graph to generalize to new objects and tasks; 2023's GraspGPT and similar work began drawing on the common sense in large language models to handle object-task combinations never seen before.

ExampleThe same knife: a robot holds it by the handle when cutting vegetables itself, but when handing it to a person, grips the back of the blade and orients the handle toward them.

Also called

TOG, Functional Grasping

Related

Grasping · Affordance · Affordance Detection · Grasp Pose Detection · Tool Use · Human-Robot Handover

Sources

Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping (Murali et al., CoRL 2020)
GraspGPT: Leveraging Semantic Knowledge from a Large Language Model for Task-Oriented Grasping

As of

2023-07