Alchemy (Deep-Learning Slang)

Chinese deep-learning slang for training models and tuning hyperparameters by trial and error, likening it to alchemy.

This is slang from China's deep-learning community that compares training a model to alchemy. Data and a model go into a GPU server (jokingly the “alchemy furnace”), you set hyperparameters — configuration choices like learning rate and batch size fixed before training starts, nicknamed the “heat” of the fire — then wait hours or days for a result whose cause is often hard to pin down, so all you can do is try again. Adjusting these hyperparameters is called “炼丹” (elixir-refining) or “调参” (tuning), and the people who do it jokingly call themselves “tuning wizards” or “alchemists.” English-speaking ML has voiced a similar complaint: in his 2017 NeurIPS Test of Time talk, Ali Rahimi said outright that “machine learning has become alchemy,” criticizing the field's reliance on techniques nobody fully understands. More systematic approaches to tuning include grid search, random search, Bayesian optimization, and population-based training.

ExampleA VLA fine-tune isn't working well, so a student changes the learning rate from 1e-4 to 2e-5, the batch size from 32 to 128, trains for 20,000 more steps, and checks again — round after round of this trial and error is what “alchemy” refers to.

Also called

Elixir Refining, Tuning Wizard (slang), Alchemist (slang)

Related

Hyperparameter · Learning Rate · Batch Size · Population-Based Training · Ablation Study · Random Seed and Reproducibility

Sources

Reflections on Random Kitchen Sinks (Ali Rahimi & Ben Recht, NIPS 2017 Test-of-Time talk)
Wikipedia: Hyperparameter optimization