Ossification

A model's weights seem to “freeze up,” unable to absorb new information even as more training data is added.

The term “ossification” comes from OpenAI's Hernandez, Kaplan, and colleagues, in their 2021 paper “Scaling Laws for Transfer”: they found that with enough fine-tuning data, a small model that had been pretrained could actually fall behind a same-sized model trained from scratch, as if pretraining had “ossified” the weights into a bad initialization that was hard to escape. Embodied AI picked up the term after Generalist AI's GEN-0, released in November 2025: when pretraining at scale on real-robot data, a 1-billion-parameter model struggled to absorb complex, diverse sensorimotor data, its weights becoming unable to learn new information over time, while 6-billion- and 7-billion-parameter models kept improving. The team describes this as the first time ossification has been observed in robotics; previously it had only been seen in language models, and only at a much smaller, tens-of-millions-of-parameters scale. It is a reminder that as data scale grows, model capacity has to keep up.

ExampleIn GEN-0's comparison, a 1-billion-parameter model shows ossification-like stagnation when faced with large, diverse sensorimotor data, while 6-billion- and 7-billion-parameter models keep improving as pretraining data grows.

Also called

Model Ossification, Weight Ossification

Related

Scaling Law · GEN-0 · Parameter Count (Model Size) · Pre-training · Transfer Learning · Catastrophic Forgetting

Sources

Generalist AI 2025-11-04: GEN-0: Embodied Foundation Models That Scale with Physical Interaction
Hernandez et al. 2021: Scaling Laws for Transfer

As of

2025-11