Curriculum-Guided Layer-Scaling for Language Model Pretraining

Karanpartap Singh1  Neil Band1  Ehsan Adeli1

1Stanford University


lm_scaling

Curriculum-Guided Layer Scaling (CGLS) is a new paradigm for compute-efficient language model pretraining that grows data complexity and model depth in tandem. We illustrate CGLS for a Llama-3.2-1B scale model with four training stages. Training begins with an 8-layer model on a data split consisting equally of data from all levels (high-school, undergraduate, and graduate). The learned weights from this stage are transferred to a larger 10-layer model, freezing the pretrained weights and training the new layers on a small, balanced data split for better initialization. The entire model is then unfrozen and pretrained on the more difficult data split. This process is repeated until the target model scale is reached.


Paper
Code

Citation

If you use this work or find it helpful, please cite us!

@misc{singh2026curriculumguidedlayerscalinglanguage,
      title={Curriculum-Guided Layer Scaling for Language Model Pretraining}, 
      author={Karanpartap Singh and Neil Band and Ehsan Adeli},
      year={2026},
      eprint={2506.11389},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2506.11389}, 
}