Navigation Error / Oracle Success Rate / Trajectory Length

The three standard Vision-and-Language Navigation metrics: how close the agent stops to the goal, whether it ever passed it, and total distance traveled.

These three metrics are a standard fixture in Vision-and-Language Navigation (VLN — following a natural-language instruction to reach a goal inside a house) results tables, used since Anderson and colleagues introduced the R2R benchmark at CVPR 2018. Navigation Error (NE) is the shortest-path distance, in meters, from where the agent finally stops to the goal; lower is better, and R2R counts an episode successful if NE is under 3 meters. Oracle Success Rate (OSR, sometimes written OS) instead asks whether the episode would count as successful if the agent had stopped at the point on its own trajectory closest to the goal — this separates “walked past the goal but didn't stop there” from “never got close at all.” Trajectory Length (TL) is the total distance actually walked; an unusually long TL signals wandering or backtracking. The three are normally reported together with Success Rate, SPL, and nDTW.

ExampleIn the original R2R paper, on a test set of previously unseen buildings, human annotators reached an 86.4% success rate while the authors' sequence-to-sequence baseline achieved only 20.4%.

Also called

NE, OSR, OS, TL, Oracle Success

Related

Vision-and-Language Navigation · Room-to-Room · Success Rate · Success weighted by Path Length · normalized Dynamic Time Warping

Sources

Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments (R2R, CVPR 2018)
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments (VLN-CE)