TL;DR. Each token's loss falls in one sigmoid-shaped step at its own learning time \(\tau\), and the steps share one pulse shape. The smooth scaling law is therefore set by when tokens are learned, the learning-time spectrum \(p(\tau)\), not by how. Reshaping the training data by learning time makes validation loss fall 11% faster.
- 110+
- pre-training runs
- 290M–6B
- parameters
- 1B–300B
- training tokens
- 1,178
- A100 GPU-days
Part 1
Real Pre-training Data
Validation loss of one million tokens in context, measured along training steps \(T\), data scale \(D\), and model scale \(M\).
1.1Token loss trajectories
measured loss sigmoid fit, coloured by \(\tau\) 10–90% of the drop
1.2From tokens to the loss curve
measured sigmoid-implied power law \(E+ka^{-\alpha}\)
measured mean mean of the sigmoid fits group's \(\tau\)
the groups of B, each shifted by its \(\tau\): one shared step
bin's \(\tau\): one pulse shape, shifted in time
\(p(\tau)\) power law \(c\,\tau^{-\beta}\)
measured sigmoid-implied power law spectrum-dominated \(p*g\)
Part 2
Synthetic Data
Mano from PhysicsLM: arithmetic expressions modulo 23 with \(L=1\) to \(10\) operations, trained with every \(L\) equally likely. The loss is measured on the answer token of 10,000 held-out problems.
2.1Problems and their trajectories
2.2Explore the mechanism
2,000 synthetic tokens, each \(\ell_i(a)=\ell_i^{\text{post}}+\Delta_i\,[1-\sigma(\lambda_i(a-\tau_i))]\). Drag across any panel to move through training. The toy illustrates the mechanism and is not fitted to data.
BibTeX
@inproceedings{wang2026smooth,
title = {Smooth Scaling Laws Hide Stepwise Token Learning},
author = {Wang, Pingjie and Hu, Zechen and Yang, Peiru and Guo, Fu and Zhang, Debing},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026},
url = {https://openreview.net/forum?id=THZDVZArRH}
}