Exploration vs. Exploitation

The trade-off between trying new actions to find something better and using the best-known action to collect reward now.

This is a fundamental tension in reinforcement learning and sequential decision-making. Exploitation means picking the action that currently looks best given what the agent already knows; exploration means trying an uncertain, unfamiliar action, which may pay off worse in the short term but could reveal a better strategy. Pure exploitation easily gets stuck on a suboptimal solution, while pure exploration never cashes in on the reward it finds. The classic framework for studying this is the multi-armed bandit problem; common strategies include ε-greedy (pick a random action with small probability ε), UCB (upper confidence bound, which favors options with more uncertainty), and Thompson sampling. Deep reinforcement learning also uses entropy regularization (encouraging the policy to stay somewhat random) and intrinsic reward (rewarding “curiosity”). Exploration is harder on real robots: unconstrained trial and error can damage the hardware, so policies are often first brought into a reasonable region using demonstration data or simulation, and only then explore in a controlled way.

ExampleWhen SimpleVLA-RL fine-tunes a VLA with reinforcement learning, it raises the rollout sampling temperature from 1.0 to 1.6 and widens the clipping range from 0.2 to 0.28, so the model generates more varied trajectories and explores more.

Also called

Exploration-Exploitation Trade-off, Exploration-Exploitation Dilemma

Related

Intrinsic Motivation · Entropy Regularization · Reinforcement Learning · Sample Efficiency · Safe Reinforcement Learning · Real-World Reinforcement Learning

Sources

Wikipedia: Exploration–exploitation dilemma
SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning