The transformer block

Attention doesn’t know about order

Here’s a surprise: self-attention treats its input as a set. Shuffle the tokens and every token gets the same mixture, just shuffled. “Dog bites man” and “man bites dog” would look identical. So transformers add a position vector to each token’s embedding before anything else.

The original transformer used fixed waves of different frequencies (each position gets a unique pattern, like the hands of a clock); many models instead simply learn a position embedding table, just like the token embeddings. Our tiny GPT does the latter.