Success weighted by Path Length

A navigation metric that credits both reaching the goal and taking a path close to the shortest possible route.

SPL was proposed by Peter Anderson, Jitendra Malik, and 9 other researchers in their 2018 paper “On Evaluation of Embodied Navigation Agents,” which recommended it as the primary metric for embodied navigation. It is computed as follows: for each test episode i, S_i is 1 if the episode succeeded and 0 otherwise; l_i is the length of the shortest path from the start to the goal; p_i is the length of the path the agent actually took; and SPL is the average, over all episodes, of S_i · l_i / max(p_i, l_i). “Success” requires the agent to actively output a stop action while close enough to the goal — the paper recommends a default threshold of twice the agent's body width — and distance is measured along the shortest path around obstacles rather than as a straight line. SPL penalizes taking a roundabout route, preventing an agent that wanders randomly and happens to reach the goal from getting a perfect score under success rate alone. Tasks such as point-goal and object-goal navigation typically report both success rate and SPL together.

ExampleThe paper gives worked examples: if half the episodes succeed and each follows the shortest path, SPL is 0.5; if all episodes succeed but each path is twice the shortest-path length, SPL is also 0.5; and if half succeed with paths twice as long, SPL is 0.25.

Also called

SPL, Success weighted by normalized inverse Path Length

Related

Success Rate · Point-Goal Navigation · Object-Goal Navigation · Navigation · normalized Dynamic Time Warping · Navigation Error / Oracle Success Rate / Trajectory Length

Sources

On Evaluation of Embodied Navigation Agents (arXiv 1807.06757)