Human Intervention Data

Data recorded when a human takes over from an autonomously running policy just as it's about to make a mistake.

Human intervention data is recorded when, during a policy's autonomous run on a real robot, a human sees it about to go wrong and takes over by teleoperation, steering the robot back on track; the observations and the human's actions during that takeover are what get recorded. The idea traces back to interactive imitation learning methods like DAgger: pure behavior cloning has only ever seen an expert's “standard route,” so once it drifts off that path it has no idea what to do, and the error compounds. Intervention data fills in exactly “how to recover once things have gone wrong,” concentrated at precisely the states where the policy is weakest. HG-DAgger (2018) has a human take over whenever they judge the situation unsafe; Berkeley's HIL-SERL uses human intervention to guide exploration during real-robot reinforcement learning; and Physical Intelligence's π*0.6 also trains on a mix of expert interventions and autonomous experience.

ExampleEarly in HIL-SERL training, a human takes over frequently, demonstrating how to complete the task from a wide range of states; once the policy's success rate rises, the amount of intervention is gradually reduced.

Also called

Takeover Data

Related

Human-in-the-Loop · Human-Gated DAgger · Recovery and Correction Data · HIL-SERL · RECAP · Intervention Rate

Sources

HG-DAgger: Interactive Imitation Learning with Human Experts (arXiv 1810.02890)
HIL-SERL 项目主页 (Chinese)
π*0.6: a VLA That Learns From Experience (arXiv 2511.14759)