Lesson 9.2MathCode~1 h

Self-attention, step by step

Queries, keys, values and the causal mask — every matrix, visible.

Module 9 · 0/8 done

All tokens at once

Attention for one query is a soft lookup. Self-attention does it for every token in the sequence at the same time, with matrices. Stack the token embeddings as the rows of $X$ :

Q = XW_Q

K = XW_K

V = XW_V

\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d}}\right) V

This single line is the heart of every modern language model. $QK^{\top}$ computes every query-key dot product in one matrix multiply; softmax (row by row) makes the weights; multiplying by $V$ blends the values.