The bigram model

A language model in a table

A language model predicts the next token from the ones before it: $P(\text{next} \mid \text{previous})$. The simplest possible version only looks at one previous token. Take ~600 first names, and for every letter, count which letter comes next. That table of counts is the model — a bigram model (bi = two: pairs of letters).

We add a special “.” token for the start and end of a name, so the model also learns how names begin and end.