Out-of-Distribution

Test-time data that comes from a different distribution than the training data, such as new objects or scenes.

Machine learning typically assumes training and test data come from the same distribution (the i.i.d. assumption). Out-of-distribution (OOD) describes test inputs that fall outside the training distribution — unseen objects, lighting, tablecloths, camera angles, or ways of phrasing an instruction; inputs that do fall within the training distribution are called in-distribution (ID). Model performance usually drops noticeably on OOD inputs, and the study of keeping models useful anyway is called OOD generalization; a 2021 survey by Peng Cui's group at Tsinghua University gives a systematic overview. Robots run into this especially often: the real world varies endlessly while demonstration data only covers a limited set of scenarios, and once execution drifts even slightly off course, later observations drift away from the training data too, a problem called distribution shift. When papers evaluate “generalization,” they are usually measuring success rate under deliberately constructed out-of-distribution conditions.

ExampleA policy trained only with a red cup on a white table is tested with a wood-grain table and a blue bowl instead — that is an out-of-distribution test.

Also called

OOD, OOD Generalization, Out-of-Distribution Generalization

Related

In-distribution · Generalization · Distribution Shift · Long-tail Problem · Robustness · Zero-shot

Sources

Towards Out-Of-Distribution Generalization: A Survey (arXiv:2108.13624)