Hand-Object Interaction

The study of how human hands contact, grasp, and manipulate objects, including estimating their joint 3D pose and contact.

Hand-object interaction is a research direction in vision and robotics focused specifically on how human hands contact, grasp, and manipulate objects — narrower than general human-object interaction, which covers the whole body. Typical tasks include jointly estimating hand pose (usually a MANO mesh) and the object's 6D pose, inferring the contact region and grasp type, recognizing what action the hand is performing, and generating plausible grasps. Representative datasets include DexYCB (CVPR 2021, which annotates MANO hand pose and object 6D pose while a hand grasps YCB objects), HOI4D (CVPR 2022, with 2.4 million frames of egocentric RGB-D video across 16 object categories and 800 instances), ARCTIC, and OakInk. For robots, human hands are a ready-made source of dexterous manipulation demonstrations: extracting hand and object trajectories from human video can be converted into dexterous-hand actions or grasp targets, and DexYCB specifically targets scenarios where an object is handed from a human to a robot.

ExampleFrom a first-person cooking video, HaMeR reconstructs the hand's 3D pose while FoundationPose tracks the spatula's pose, giving the position of the hand gripping the handle and the stir-fry trajectory — which is then retargeted into demonstration data for a dexterous hand.

Also called

HOI (Hand-Object)

Related

Hand Pose Estimation · 6D Object Pose Estimation · DexYCB · HOI4D · ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation · Human Video Data

Sources

HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction (arXiv:2203.01577)
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (arXiv:2104.04631)