Gradient
Lesson 9.8ProjectModelCode~2 h 30 min

Project: your own GPT

Train a transformer on text you choose, watch for overfitting with a validation split, and tune it.

Module 9 · 0/8 done

Project: train your own GPT

This is the big one: a complete, honest language-model training run, built from parts you understand. The model is the GPT from this module. Everything around it is yours: the tokenizer, the data split, the schedule, clipping, validation, sampling.

  1. Tokenize your text, split it, and set a baseline.
  2. Train with warm-up, cosine decay and gradient clipping, tracking validation loss.
  3. Generate text with temperature and top-k, and measure perplexity.