Attention, intuitively

What does “it” mean?

Read this: “The animal didn’t cross the street because it was too tired.” What does “it” refer to? The animal. Now change one word: “…because it was too wide.” Now “it” is the street. To understand a word, you have to look at other words — and which other words matters depends on the whole sentence.

Our neural language model from Module 8 had no way to do this: it glued a fixed window of letters together and treated every position the same. Attention gives every token the ability to look back at all the others and decide, for itself, which ones are relevant.