Floating-Point Operations (FLOPs)

The number of floating-point additions and multiplications a computation takes — a measure of how computationally heavy a model is.

FLOPs stands for floating-point operations: the total number of floating-point arithmetic operations a computation requires, used as a measure of a model's computational cost. The lowercase ‘s’ just marks the plural, which is a different thing from the uppercase FLOPS (‘per second’), a measure of hardware compute capacity. There is a widely used Transformer estimate, from OpenAI's 2020 scaling-law paper: processing one token in a forward pass through a model with N parameters takes about 2N floating-point operations, and about 6N once training's backward pass is included; total training compute is also often reported in FLOPs. For robots, FLOPs matters because it drives inference latency directly: on the same onboard chip, a policy with more FLOPs runs slower and forces a lower control frequency, which is why on-device deployment usually relies on shrinking the model and pruning visual tokens to cut FLOPs.

ExampleUsing the 2N rule of thumb, a 7-billion-parameter VLA processing a single token costs about 14 billion floating-point operations (14 GFLOPs) in its forward pass; feeding in 256 image tokens at once costs roughly 3.6 trillion operations for that part alone (not counting attention).

Also called

FLOPs, FLOP

Related

Parameter Count (Model Size) · Scaling Law · FLOPS / TFLOPS (Floating-Point Operations per Second) · Inference Latency · On-device Model · Visual Token Pruning

Sources

Scaling Laws for Neural Language Models (Kaplan et al., 2020)
Scaling Laws for Neural Language Models（ar5iv 全文，含 C≈6N 推导） (Chinese)