NeurIPS 2026 Spotlight

Smooth Scaling Laws Hide
Stepwise Token Learning

Pingjie Wang1,2,*,‡ Zechen Hu1,*,† Peiru Yang1,3,*,‡ Fu Guo1 Debing Zhang1,†

1Dots Studio, Xiaohongshu Inc. 2Shanghai Jiao Tong University 3Tsinghua University

*Equal contribution †Corresponding authors ‡Work done during an internship at Dots Studio, Xiaohongshu Inc.

TL;DR. Each token's loss falls in one sigmoid-shaped step at its own learning time \(\tau\), and the steps share one pulse shape. The smooth scaling law is therefore set by when tokens are learned, the learning-time spectrum \(p(\tau)\), not by how. Reshaping the training data by learning time makes validation loss fall 11% faster.

110+
pre-training runs
290M–6B
parameters
1B–300B
training tokens
1,178
A100 GPU-days

Part 1

Real Pre-training Data

Validation loss of one million tokens in context, measured along training steps \(T\), data scale \(D\), and model scale \(M\).

1.1Token loss trajectories

measured loss sigmoid fit, coloured by \(\tau\) 10–90% of the drop

1.2From tokens to the loss curve

hover any panel: the others follow

measured sigmoid-implied power law \(E+ka^{-\alpha}\)

measured mean mean of the sigmoid fits group's \(\tau\)

the groups of B, each shifted by its \(\tau\): one shared step

bin's \(\tau\): one pulse shape, shifted in time

\(p(\tau)\) power law \(c\,\tau^{-\beta}\)

measured sigmoid-implied power law spectrum-dominated \(p*g\)

Part 2

Synthetic Data

Mano from PhysicsLM: arithmetic expressions modulo 23 with \(L=1\) to \(10\) operations, trained with every \(L\) equally likely. The loss is measured on the answer token of 10,000 held-out problems.

2.1Problems and their trajectories

2.2Explore the mechanism

a = 100
α = 1.20
width = 4.0
one token each, lit once learned average loss \(L\)
share learned per interval \(p(\tau)\propto\tau^{-\alpha}\)
\(-\mathrm{d}L/\mathrm{d}a\) \(p(\tau)\) × mean drop

2,000 synthetic tokens, each \(\ell_i(a)=\ell_i^{\text{post}}+\Delta_i\,[1-\sigma(\lambda_i(a-\tau_i))]\). Drag across any panel to move through training. The toy illustrates the mechanism and is not fitted to data.

BibTeX

@inproceedings{wang2026smooth,
  title     = {Smooth Scaling Laws Hide Stepwise Token Learning},
  author    = {Wang, Pingjie and Hu, Zechen and Yang, Peiru and Guo, Fu and Zhang, Debing},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://openreview.net/forum?id=THZDVZArRH}
}