SimplerEnv

A simulated evaluation suite that replicates common real-robot manipulation setups for cheap policy scoring.

SimplerEnv (paper name: SIMPLER) was proposed in 2024 by researchers at UC San Diego, Stanford, UC Berkeley, and Google DeepMind, published at CoRL 2024. It replicates two widely used real-robot setups inside the SAPIEN/ManiSkill simulator: the Google Robot platform used to collect Google's RT-1 dataset, and the WidowX arm used with the Bridge dataset — so a policy trained on real-robot data can be scored without ever touching the real robot. To narrow the sim-to-real gap, it calibrates controller parameters and offers two evaluation modes: “visual matching” (compositing simulated objects onto a real background image) and “variant aggregation” (building several simulated variants with different backgrounds, lighting, and tabletop textures, then averaging the results). The authors verified that simulated and real-robot rankings of methods correlate closely, and many VLA papers since have reported success rate on it.

ExampleTo evaluate a WidowX policy, run tasks like “put the spoon on the towel,” “put the carrot on the plate,” and “put the eggplant in the basket” directly in SimplerEnv and tally success rate, without arranging objects by hand on a real robot each time.

Also called

SIMPLER: Simulated Manipulation Policy Evaluation for Real Robot Setups, SIMPLER

Related

Simulation-Based Evaluation · Visual Matching (SimplerEnv) · Variant Aggregation (SimplerEnv) · Mean Maximum Rank Violation · Sim-to-Real Correlation · ManiSkill

Sources

Evaluating Real-World Robot Manipulation Policies in Simulation (SIMPLER, arXiv 2405.05941)
SIMPLER 项目主页 (Chinese)
simpler-env/SimplerEnv (GitHub)

As of

2026-09