Emergent Abilities

Abilities absent in small models that only appear once model or data scale grows large enough.

This idea was systematically laid out in a 2022 paper by Jason Wei and colleagues, “Emergent Abilities of Large Language Models,” which defines an emergent ability as one that is absent in smaller models, appears in larger models, and cannot be predicted by extrapolating from smaller models' performance — multi-step arithmetic and chain-of-thought reasoning are examples. In 2023, Rylan Schaeffer and colleagues pushed back, arguing that much of this apparent “sudden appearance” is an artifact of the evaluation metric: metrics that only score an all-or-nothing correct answer look like a sharp jump, but switching to a continuous metric shows the improvement is actually smooth. In embodied AI, “emergent” usually refers to a capability that was not in the training data but shows up at test time anyway — for example, RT-2 following instructions absent from any robot data, such as placing an object on a specific number or icon, or picking up a rock to use as an improvised hammer. Claims of emergence in marketing materials should be checked against the actual evaluation behind them.

ExampleRT-2's robot demonstration data contains no task like “put the object on the numbered card,” but its vision-language-model backbone recognizes digits from web image-text data, so it can follow the instruction anyway.

Also called

Emergence, Emergent Capabilities

Related

Scaling Law · Large Language Model · Chain-of-Thought · RT-2 · Zero-shot · Generalization

Sources

Emergent Abilities of Large Language Models (arXiv 2206.07682)
Are Emergent Abilities of Large Language Models a Mirage? (arXiv 2304.15004)
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (arXiv 2307.15818)