Web IR NLP Group @ NUS
Web IR NLP Group @ NUS
Tour
News
People
Events
Gallery
Publications
Projects
Contact
Light
Dark
Automatic
Long Context
Context Window Scheduling for Efficient LLM Pretraining
Longer pretraining context windows are not always better. We study how to schedule the context window during LLM pretraining to balance long-context capability with training efficiency.
Tongyao Zhu
,
Qian Liu
,
Min-Yen Kan
Paper
arXiv
Code
Slides
Cite
×