Learning track
Pretraining at Scale
Design the objectives, data, optimization, parallelism, and cost model behind a serious language-model pretraining run.
Pretraining at Scale
Design the objectives, data, optimization, parallelism, and cost model behind a serious language-model pretraining run.
- 06.01Objectives: causal LM, MLM, span corruption, FIM→
- 06.02Data: sourcing, filtering, dedup, decontamination→
- 06.03Training your own tokenizer→
- 06.04Scaling laws I: Kaplan→
- 06.05Scaling laws II: Chinchilla & compute-optimality→
- 06.06Scaling laws III: inference-aware & over-training→
- 06.07Data parallelism, ZeRO & FSDP→
- 06.08Tensor, pipeline & sequence parallelism→
- 06.09Mixed precision: fp16, bf16, fp8→
- 06.10Gradient checkpointing & memory maths→
- 06.11Learning-rate schedules & warmup→
- 06.12When training goes wrong: spikes, divergence, NaNs→
- 06.13What a real pretraining run costs→