Train a tiny GPT

A real transformer, in your tab

Below is a complete GPT-style language model: token and position embeddings, two transformer blocks with four attention heads each, layer norms, residual connections, GELU MLPs, and a softmax over characters. About 60,000 parameters. It’s built entirely from the Tensor engine in this course, and it trains on a small collection of simple stories — on your device, in a background thread.

Press Start training and watch. At first it writes noise. Within seconds it discovers spaces and common letters. Then short words. Then whole phrases.