CMU Advanced NLP Spring 2025 (6): Pretraining

This lecture (by Sean Welleck) for CMU CS 11-711, Advanced NLP covers:
Major factors in pretraining
Pretraining objectives
Data: quantity, quality, and coverage
Compute and scaling laws

[@prashlovessamosa]

Thanks 👍

[@420_gunna]

Thanks for these, Sean! Small nit on your course site -- for the 1/30/2025 class, the link for Language Models are Unsupervised Multitask Learners (Radford et al 2019) actually points to Sasha Rush's The Annotated Transformer article.