Scaling laws in NLP

Beginner

Scaling laws say that if you train language models in a reasonably consistent way, performance usually improves along a smooth curve as you increase size and training resources. The headline is not simply "bigger is better". The real lesson is that parameters, data, and compute must scale together if you want efficient gains.

What actually scales?

• Model size: more parameters usually reduce language modeling loss.
• Dataset size: more training tokens help, especially when the model is large enough to use them.
• Compute budget: training FLOPs limit how large a model you can train and how many tokens you can afford.
• Loss or error: the quantity being improved, often cross-entropy loss during pretraining.

Power-law intuition

One of the classic findings is that loss often falls roughly like a power law. Informally, every large increase in resources produces a smaller but still somewhat predictable improvement. That is why a 10x increase in model size does not produce a 10x quality jump, yet it can still be strategically worth it.

Very rough intuition

More parameters  -> lower loss
More data        -> lower loss
More compute     -> lower loss

But each extra step usually buys less than the previous one.
That is diminishing returns, not random behavior.

Why the topic matters

• It helps estimate whether the next training run is likely to be worthwhile.
• It explains why very large models can be more sample-efficient than small ones.
• It helps avoid waste, such as training a huge model on too little data or a small model on far too much data.
• It supports planning before expensive experiments are launched.

Simple example

runs = [
    {"params_b": 0.3, "tokens_b": 20, "loss": 2.10},
    {"params_b": 1.3, "tokens_b": 50, "loss": 1.82},
    {"params_b": 6.7, "tokens_b": 200, "loss": 1.55},
]

for run in runs:
    print(run)

The pattern to notice is not the exact numbers. It is the shape: when resources grow in a coordinated way, loss tends to improve smoothly enough that future runs can often be forecast from smaller pilot runs.

Advanced

Advanced discussion focuses on how to allocate a fixed compute budget. Early large-scale work showed that language model loss follows clean power-law trends across model size, dataset size, and compute over many orders of magnitude. Later work revised the practical recipe: many famous models were too large for the amount of data they saw, meaning they were undertrained relative to their compute budget.

Canonical scaling-law view

• Kaplan-style result: loss improved smoothly with larger models, more data, and more compute, and larger models were often more sample-efficient.
• Compute-optimal implication: for a fixed training budget, there is an optimal balance between model parameters and number of training tokens.
• Chinchilla-style revision: many frontier models were oversized relative to their token budgets; better results came from somewhat smaller models trained on much more data.
• Practical lesson: being "largest by parameter count" is not the same as being best-trained.

Three common regimes

1. Parameter-limited
   Model is too small to absorb more data efficiently.

2. Data-limited
   Model is large, but token budget is too small, so capacity is left unused.

3. Compute-limited
   You know both would help, but available training FLOPs force a trade-off.

Compute-optimal training in plain language

If you double parameters but do not materially increase tokens, the model may be too large for the amount of learning signal it receives. If you only add data while keeping the model too small, the model may not have enough capacity to exploit that data. Compute-optimal training tries to balance these two inefficiencies.

A commonly cited later takeaway is that model size and token count should often grow together much more than earlier practice assumed. This shifted the field away from extreme parameter growth with relatively fixed data budgets.

A compact mathematical sketch

# Stylized loss decomposition, not a production formula.
# Irreducible loss + model term + data term

def estimated_loss(irreducible, params, tokens, a=150, b=1.0, alpha=0.34, beta=0.28):
    model_term = a * (params ** (-alpha))
    data_term = b * (tokens ** (-beta))
    return irreducible + model_term + data_term

print(round(estimated_loss(1.45, params=70e9, tokens=1.4e12), 4))

The important concept is the shape of the formula, not the constants. As parameters and tokens grow, the added benefit shrinks, and the curve approaches an irreducible floor set by the task and data distribution.

What scaling laws are good for

• Forecasting: estimate expected gains before training the full model.
• Run design: choose whether extra budget should go toward more parameters or more tokens.
• Interpreting benchmark jumps: a capability increase may come from smoother loss improvements rather than a mysterious phase change.
• Efficiency analysis: compare the marginal value of the next scaling step against its cost.

Batch size in scaling

Batch size is a key variable in compute-optimal training. Scaling laws treat compute as the primary budget, and batch size directly controls how that compute is spent per step.

• Critical batch size: for a given model and loss level, there exists a critical batch size below which training is compute-bound (more steps, each fast) and above which it is data-bound (fewer steps, each expensive but with diminishing returns). Operating near the critical batch size maximises compute efficiency.
• Scaling batch size with model size: larger models generally tolerate and benefit from larger batch sizes. GPT-3 used effective batch sizes in the millions of tokens.
• Batch size vs. tokens seen: doubling the batch size halves the number of parameter updates for the same number of tokens. Beyond the critical batch size, this trade-off stops being worthwhile — each update becomes less informative.

Below critical batch size:
  Each step is cheap → many steps → compute-efficient

Above critical batch size:
  Each step is expensive → few steps → wasted compute per step
  Diminishing gradient quality improvement

Optimal: operate near the critical batch size for the current loss level
				

Limits and caveats

• Scaling laws are empirical regularities, not laws of nature. They depend on architecture family, optimizer, data mixture, and training recipe.
• Cross-entropy loss is easier to scale smoothly than many downstream behaviors users care about.
• Data quality matters. More low-quality or duplicated data can weaken the expected gain.
• Extrapolating too far beyond observed runs is dangerous, especially when the recipe changes.
• Post-training methods can alter downstream performance without changing the original pretraining scaling curve.

Planning question for a fixed compute budget

Option A: much larger model + too few tokens   -> likely undertrained
Option B: balanced model + balanced token set  -> often compute-optimal
Option C: small model + massive token set      -> likely capacity-limited

Scaling-law analysis helps choose B on purpose rather than by guesswork.