What is Scaling Law?
Also called neural scaling law.
A scaling law is an empirical relationship describing how a model's loss improves as a power function of model size, training data volume, and compute. These relationships have held across many orders of magnitude, allowing researchers to predict the performance of a large training run from much smaller experiments. They guide how a fixed compute budget is split between parameters and data.
The core finding is that test loss falls smoothly and predictably as any of the three inputs increases, with the others held sufficient. Plotted on logarithmic axes the relationship is close to a straight line. This regularity is remarkable because nothing in the theory of neural networks required it, and it turned large-scale training from guesswork into budget planning.
The practical question is allocation. Given a fixed compute budget, a run can use more parameters on less data or fewer parameters on more data. Influential work revised earlier practice by showing that many large models had been undertrained relative to their size, and that scaling parameters and tokens together yields better loss for the same compute.
Inference economics complicate the optimum. A compute-optimal training recipe minimizes loss per unit of training compute, but a model that will serve many requests benefits from being smaller and trained longer, since every request pays for its size. Production models are therefore commonly trained well past the compute-optimal data point on purpose.
Scaling laws describe loss, not capability, and they have limits. They say nothing directly about instruction following, factual accuracy, or safety. High-quality data appears to shift the curves rather than merely move along them, and the supply of high-quality text is finite, which is why data curation, filtering, and synthetic generation now receive as much attention as raw scale.
Key points
- Loss falls predictably with size, data, and compute
- Lets large run outcomes be forecast from small experiments
- Guides how to split a budget between parameters and tokens
- Serving cost favors smaller models trained longer
- Predicts loss, not usefulness, safety, or accuracy
In practice
A lab planning a major training run first trains a ladder of small models at increasing compute budgets and fits the resulting loss curve. Extrapolating tells them roughly what loss the full run should reach and how to divide the budget between parameter count and training tokens. During the real run they compare observed loss against the prediction; a persistent gap signals a data or optimization problem worth halting for.