ARNOLD

A benchmark in Isaac Sim testing whether a robot can manipulate objects to a specified continuous state based on language.

ARNOLD was proposed by the Beijing Institute for General Artificial Intelligence (BIGAI) together with UCLA, Peking University, Tsinghua, and Columbia, published at ICCV 2023. Earlier language-conditioned manipulation benchmarks mostly treated the goal as a binary state like open/closed; ARNOLD instead requires reaching a continuous target value, such as pulling a drawer open to a specified degree or pouring out a specified proportion of water, counting success only when the object's state stays within a tolerance range around the target. It is built on NVIDIA Isaac Sim and PhysX 5.0, and includes 8 tasks (picking up an object, adjusting orientation, opening/closing a drawer, opening/closing a cabinet door, pouring water, transferring water), 40 object types, 20 scenes, and 10,000 expert demonstrations, with language instructions generated from templates. Besides in-distribution testing, evaluation separately reports generalization splits for novel objects, novel scenes, and novel target states; the authors found that the language-conditioned policies available at the time performed noticeably worse on these splits.

ExampleGiven an instruction to open a cabinet door halfway, the robot must not only open the door but also stop its opening angle within an allowed range around the target value to count as successful.

Also called

A Benchmark for Language-Grounded Task Learning with Continuous States in Realistic 3D Scenes

Related

Language-conditioned Policy · Articulated Object Manipulation · NVIDIA Isaac Sim · PhysX · Generalization / Robustness Evaluation · Benchmark

Sources

ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes (arXiv 2304.04321)
ARNOLD project page