(no title) – Karan P. Singh

Curriculum-Guided Layer-Scaling for Language Model Pretraining

Karanpartap Singh¹ Neil Band¹ Ehsan Adeli¹

¹Stanford University

Curriculum-Guided Layer Scaling (CGLS) is a new paradigm for compute-efficient language model pretraining that grows data complexity and model depth in tandem. We illustrate CGLS for a Llama-3.2-1B scale model with four training stages. Training begins with an 8-layer model on a data split consisting equally of data from all levels (high-school, undergraduate, and graduate). The learned weights from this stage are transferred to a larger 10-layer model, freezing the pretrained weights and training the new layers on a small, balanced data split for better initialization. The entire model is then unfrozen and pretrained on the more difficult data split. This process is repeated until the target model scale is reached.

Paper

Code

Citation

If you use this work or find it helpful, please cite us!

@misc{singh2026curriculumguidedlayerscalinglanguage,
      title={Curriculum-Guided Layer Scaling for Language Model Pretraining}, 
      author={Karanpartap Singh and Neil Band and Ehsan Adeli},
      year={2026},
      eprint={2506.11389},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2506.11389}, 
}