Lesson 9.6ConceptMath~45 min

Training at scale

Where the memory and compute go, warm-up schedules, gradient clipping, and splitting the work across thousands of chips.

Module 9 · 0/8 done

From your laptop to a data centre

The model you trained in the tiny-GPT lesson had 60 000 parameters and took a minute. Frontier models have hundreds of billions of parameters and train for months on thousands of GPUs. The ideas are the same ones you know. What changes is engineering: keeping a gigantic, expensive run stable, and fitting it into hardware at all.

Fig 9.8Pick a model size. Training memory is about 16 bytes per parameter before you even store the activations. Then pick how much data to train on, and see the compute bill.