Distribution Shift

When the data seen at deployment has a different distribution than the training data, hurting performance.

Distribution shift broadly describes any mismatch between the distribution of training data and the distribution of data seen in actual use. Covariate shift is one specific kind: the distribution of inputs changes, but the true mapping from input to correct output stays the same; two other common kinds are label shift and concept shift. This is especially prominent in robot imitation learning: a policy only ever sees states the expert visited during training, so the moment it makes even a small error during execution, it drifts into states the demonstrations never covered, and that error compounds into a bigger deviation, a problem called compounding error. Stéphane Ross and colleagues, in a 2011 AISTATS paper, pointed out that this setting, where the agent's own actions determine its next input, violates the i.i.d. assumption, and proposed DAgger: let the policy run on its own, have an expert label the correct action for the new states it encounters, and gradually pull the training distribution toward the distribution the policy actually meets.

ExampleA cup-grasping policy trained with behavior cloning always approaches the cup from directly above in the demonstrations. At deployment the arm is off by 2 cm, a position never seen in training, so the policy's output gets progressively worse until it knocks the cup over.

Also called

Covariate Shift, Dataset Shift

Related

Compounding Error · DAgger · Behavior Cloning · Out-of-Distribution · Recovery and Correction Data · Domain Adaptation

Sources

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (arXiv 1011.0686)
Domain adaptation - Wikipedia