Cherry-picking

cherry-pick（挑结果）Common

Showing only the best-looking results out of many attempts while hiding failures and the average performance.

The term originally means picking only the best cherries, and in research and marketing it means showing only the successful or most impressive runs out of many attempts. A robot policy's performance can vary a lot from one trial to the next — the same task might succeed 3 times out of 10 — so releasing only the successful clips can make people overestimate its ability. To spot this, check whether the total number of trials and the success rate are reported, whether the test objects and settings differ from training, and whether the video is an unbroken, single continuous take. Rigorous papers report success rates, confidence intervals, and failure cases.

ExampleA paper's project page shows only 3 successful clips of a robot folding clothes, with no mention in the text of how many total attempts were made or what the overall success rate was.

Also called: Cherry-pick
Related: Edited Demo · Demo (Demonstration Video) · Success Rate · One-Take (Uncut) Video · Leaderboard Chasing
Sources: Cherry picking - Wikipedia