Context window scheduling (ours) versus constant long-context pretraining and continual long-context pretraining.Recent advances in LLM pretraining have featured ever-expanding context windows to process longer sequences. Long context windows are costly to train with, however, and it is not obvious that every token should be trained at the full window length. This project, carried out with collaborators at Sea AI Lab, studies how the context window itself should be scheduled during pretraining.
In SkyLadder: Better and Faster Pretraining via Context Window Scheduling (NeurIPS 2025), a pilot study shows that under a fixed token budget, models pretrained with shorter context windows consistently outperform their long-context counterparts. Motivated by this, SkyLadder implements a simple short-to-long context window transition during pretraining. Pretraining 1B-parameter models (up to 32K context) and 3B-parameter models (8K context) on 100B tokens, SkyLadder yields consistent gains of up to 3.7% on common benchmarks and matches or exceeds baseline results on long-context tasks, while training up to 22% faster than the baselines. The SkyLadder repository provides the pretraining code.
