ObjectFolder

A multisensory object dataset providing an object's appearance, tapping sound, and tactile readings all together.

ObjectFolder is a series of multisensory object datasets released by Jiajun Wu, Fei-Fei Li, and others at Stanford. Version 1.0 (CoRL 2021) contains 100 virtual objects, using implicit neural representations (a small network that stores one kind of sensory data and outputs it on query) to encode vision, sound, and touch in a unified way; version 2.0 (CVPR 2022) expanded to 1,000 objects and improved rendering speed and quality; ObjectFolder Real, from 2023, captured meshes, video, tapping sounds, and tactile readings for 100 real household objects, along with an evaluation benchmark of 10 tasks. It fills a gap in most object datasets, which lack sound and touch entirely, and can be used for research on cross-sensory retrieval, contact localization, shape reconstruction, and sim-to-real transfer.

ExampleGiven a rendered image of a cup, retrieve the sound it makes when tapped, or the tactile image recorded by a vision-based tactile sensor pressing on it — an example of cross-sensory retrieval.

Also called

ObjectFolder 2.0, ObjectFolder Real, ObjectFolder Benchmark

Related

Multimodal Perception · Tactile Data · Visuo-Tactile Fusion · Tactile Simulation · YCB Object and Model Set · Sim-to-Real Transfer

Sources

ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations (arXiv 2109.07991)
ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer (arXiv 2204.02389)
The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects (arXiv 2306.00956)

As of

2023-06