Definition
Scaling laws are empirical relationships fitted to measurements of how model loss or performance changes with resources. Common resource axes include parameter count, training tokens, training compute, and inference-time compute. A power law appears as a straight trend on suitable logarithmic axes, which makes it useful for comparing runs and projecting within a measured regime.
A fitted curve is conditional on the model family, data distribution, objective, optimizer, tokenization, and resource accounting used to produce it. Parameter count alone is not a complete scale variable. A larger model trained on too little data may perform worse than a smaller model trained for longer under the same compute budget.
Language-model studies
Jared Kaplan and colleagues' 2020 paper "Scaling Laws for Neural Language Models" measured power-law relationships among cross-entropy loss, model size, dataset size, and training compute. Jordan Hoffmann and colleagues revisited compute-optimal allocation in 2022. Their Chinchilla study found that, under its setup, model size and training tokens should grow together more closely than the earlier allocation suggested. The later result did not invalidate scaling analysis; it showed that conclusions change when the fitted regime and optimization question change.
Scaling laws can guide experiment budgets, but an extrapolation remains a forecast. Confidence intervals, residuals, changes in architecture, data exhaustion, and measurement error matter. Downstream capabilities may be noisy even when aggregate loss follows a smooth curve.
Distinguish it from nearby terms
A scaling law is not a universal law of intelligence. Benchmark scaling measures task scores rather than training loss. Test-time scaling changes inference resources after training. Hardware scaling measures system throughput or cost. The same word "scaling" does not make these curves interchangeable.
Check your understanding
A team fits a loss curve to dense Transformer runs between 100 million and 7 billion parameters, then forecasts a sparse MoE at 500 billion parameters. Which changes put the forecast outside its evidence, and what intermediate runs would test the extrapolation?