Gradient
Lesson 8.5ModelCode~1 h 25 min

A neural language model

Swap the count table for a network that sees several letters at once — and train it here.

Module 8 · 0/8 done

More context, smarter guesses

The bigram model only sees one letter. Seeing three would help enormously — after “emm”, the next letter is very likely “a”. But a count table for 3 letters of context needs 27 × 27 × 27 × 27 ≈ half a million cells, most of them never seen even once. Counting doesn’t scale.

In 2003, Yoshua Bengio and colleagues proposed a neural network instead. Each context letter is turned into an embedding; the embeddings are joined together and passed through a hidden layer; a softmax at the end gives the probability of every possible next letter.

eembed10 numbersmembed10 numbersmembed10 numbersconcat30 numbershidden · tanh64 neuronssoftmax27 probabilitiesnext: “a” 61%
Fig 8.5The architecture: three context letters → embeddings → one long vector → hidden layer → probabilities for the next letter.
h=tanh⁡(W1[ Ec1; Ec2; Ec3 ]+b1)\mathbf{h} = \tanh(W_1 [\,E_{c_1};\, E_{c_2};\, E_{c_3}\,] + b_1)
P(next)=softmax(W2 h+b2)\violet{P}(\text{next}) = \text{softmax}(W_2\,\mathbf{h} + b_2)