Lesson 9.7ConceptMath~55 min
Scale
Why bigger models trained on more data keep getting better — and how predictably.
Module 9 · 0/8 done
From our toy to the frontier
Architecturally, the model you just trained is a close cousin of the largest language models. What changes is scale — by staggering factors:
| model | parameters | training data | training |
|---|---|---|---|
| Our tiny GPT | 60 thousand | 3 thousand characters | seconds, one laptop |
| GPT-2 (2019) | 1.5 billion | ~10 billion tokens | days, dozens of chips |
| GPT-3 (2020) | 175 billion | ~300 billion tokens | weeks, thousands of GPUs |
| Frontier models (2025–26) | hundreds of billions to trillions | tens of trillions of tokens | months, tens of thousands of GPUs |