Intermediate Representation

Transitional information a model predicts between an instruction and low-level action, such as a 2D trajectory or keypoints.

In robot learning, an intermediate representation means not going directly from an image and instruction to joint commands, but first producing something human-readable and less tied to a specific robot, which a downstream policy or controller then turns into an action. Common forms include an end-effector trajectory drawn on the image, object keypoints, an affordance heatmap (marking where to grasp or push), a subgoal image, and a language subtask. Google DeepMind and colleagues' 2023 RT-Trajectory uses a rough trajectory sketch as the policy's condition, letting it complete new tasks that language conditioning alone couldn't; 2025's HAMSTER has a high-level vision-language model predict a 2D path, which a lower-level, 3D-aware policy then executes, reaching about 20 percentage points higher average success than OpenVLA in real-robot experiments. The benefit is that the high level can be trained on cheap data such as action-free video and simulation, and it's easier for a person to inspect and correct; the cost is that information gets compressed, and a poorly chosen representation limits precision. Compiler design has a same-named concept (IR) with a different meaning.

ExampleA language-conditioned policy trained only on pick-and-place data usually can't learn a new task like folding; RT-Trajectory instead conditions on a trajectory sketch drawn over the image, either hand-drawn or produced by a generative model, letting the policy carry out motions it never saw during training.

Also called

Mid-level Representation

Related

Hierarchical Architecture · RT-Trajectory · Affordance · Semantic Keypoints · Visual Prompting · Dual-System Architecture (System 1 / System 2)

Sources

RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches (arXiv:2311.01977)
HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation (arXiv:2502.05485)

As of

2025-05