Context Window Scheduling for Efficient LLM Pretraining

Context window scheduling (ours) versus constant long-context pretraining and continual long-context pretraining.

Recent advances in LLM pretraining have featured ever-expanding context windows to process longer sequences. Long context windows are costly to train with, however, and it is not obvious that every token should be trained at the full window length. This project, carried out with collaborators at Sea AI Lab, studies how the context window itself should be scheduled during pretraining.

In SkyLadder: Better and Faster Pretraining via Context Window Scheduling (NeurIPS 2025), a pilot study shows that under a fixed token budget, models pretrained with shorter context windows consistently outperform their long-context counterparts. Motivated by this, SkyLadder implements a simple short-to-long context window transition during pretraining. Pretraining 1B-parameter models (up to 32K context) and 3B-parameter models (8K context) on 100B tokens, SkyLadder yields consistent gains of up to 3.7% on common benchmarks and matches or exceeds baseline results on long-context tasks, while training up to 22% faster than the baselines. The SkyLadder repository provides the pretraining code.

Tongyao Zhu
Tongyao Zhu
IPP Doctoral Student (Jan ‘23; SEA)
Co-Supervised by Qian Liu.
Research Achievement Award (RAA, ‘25)

PhD Candidate January 2023 Intake

Qian Liu
Qian Liu
Research Collaborator

Researcher at TikTok AI Innovation Center; Head of Code Research Team

Min-Yen Kan
Min-Yen Kan
Associate Professor

WING lead; interests include Digital Libraries, Information Retrieval and Natural Language Processing.