Lesson 5.6MathCode~45 min

Surprise & cross-entropy

Measuring how wrong a prediction is — the loss that trains every language model.

Module 5 · 0/9 done

Measuring surprise

If something you thought was 99% likely happens, you’re not surprised. If something you thought was 1% likely happens, you’re very surprised. Information theory turns this into a number:

\pink{\text{surprise}} = -\ln(\violet{p})

The logarithm of a probability is negative (probabilities are below 1), so the minus sign makes surprise positive. And it has exactly the right shape: certain events (p = 1) have zero surprise; impossible-seeming ones explode.

Fig 5.10Drag the probability the model assigned to what actually happened. Above about 0.5 the surprise is gentle; near zero it shoots up.

Check yourself

A weather model said rain was 10% likely, and then it rained. Its surprise is −ln(0.1). Roughly how big is that?

Complete the challenge above to continue.