Lesson 8.6ModelCode~1 h

Recurrent networks

A network with memory: read text one step at a time, carrying a hidden state. Why it struggled, and what replaced it.

Module 8 · 0/8 done

A network with a memory

The neural language model reads a fixed window of 3 characters. Anything earlier is simply invisible. Text needs more: whether “bank” means a river or money can depend on words much further back.

A recurrent neural network (RNN) reads one token at a time and carries a hidden state forward: a vector that summarises everything read so far. At each step it mixes the new input with its old state:

\mathbf{h}_t = \tanh(\mathbf{x}_t W_x + \mathbf{h}_{t-1} W_h + \mathbf{b})

The same weights are used at every step, so it can read sequences of any length.

Fig 8.7The RNN unrolled: one column per letter. Each hidden state is built from the letter below it and the state to its left. Type your own word.