Encoders, decoders & BERT

Three shapes of transformer

Your tiny GPT is one kind of transformer: a decoder. Every token may only look backwards, because its job is to predict what comes next. But the same blocks can be wired up in other ways, and the difference is mostly which tokens may attend to which, set by a mask.

• Decoder-only (GPT, Llama and most chat models): a causal mask, trained to predict the next token. Built to write.
• Encoder-only (BERT): no mask at all, so every token sees the whole text, in both directions. Trained to fill in blanks. Built to understand: search, classification, embeddings.
• Encoder–decoder (the original 2017 Transformer, T5): an encoder reads the input, and a decoder writes the output while looking back at the encoder through cross-attention. Built to transform one sequence into another, as in translation.