ProgPrompt

A method that prompts a large language model with Python-like code to generate robot task plans the robot can actually execute.

ProgPrompt was released by Ishika Singh, Animesh Garg, and colleagues at the University of Southern California and NVIDIA in September 2022, published at ICRA 2023 with an extended version in Autonomous Robots. Asking a large language model (LLM) to write a plan directly in natural language often produces actions the robot cannot perform or references to objects that are not actually in the scene. ProgPrompt instead writes the prompt as code: it opens with import statements listing the available actions, gives the list of objects present in the scene, and provides a few example tasks written as Python functions (the function name is the task, the body is the steps), then has the LLM continue by writing the function for a new task. Steps in the generated plan are grouped with comments, similar to chain-of-thought reasoning, and assert statements check preconditions and trigger recovery actions when they fail. ProgPrompt achieved the best results of its time in the VirtualHome household simulator and was also validated on tabletop tasks with a real robot arm; it belongs to the same early wave of 'LLM as task planner' work as SayCan and Code as Policies.

ExampleThe prompt begins with 'from actions import walk, grab, putin, open, close', lists the scene's objects such as salmon and microwave, and gives one example function, 'throw_away_lime()'; the LLM then continues by writing out 'microwave_salmon()' step by step.

Also called

ProgPrompt: Generating Situated Robot Task Plans using Large Language Models

Related

LLM-based Task Planning · SayCan · Code as Policies · Inner Monologue · VirtualHome: Simulating Household Activities via Programs · Prompt / Prompt Engineering

Sources

arXiv 2209.11302: ProgPrompt
ProgPrompt 项目主页 (Chinese)

As of

2023-05