Lesson 5.5MathCode~40 min

Softmax: scores → probabilities

Turn any list of numbers into a probability distribution, and control its confidence.

Module 5 · 0/9 done

From scores to probabilities

A classifier looking at a photo produces one score per possible answer — cat 2.0, dog 1.0, bird 0.1, fish −1.0. These raw scores (called logits) can be any number: negative, huge, whatever. But we want probabilities: every one positive, all adding up to exactly 1.

Softmax does it in two moves. First, make everything positive by raising $e$ to the power of each score ( $e^x$ is always positive, and bigger scores stay bigger). Then divide each by the total, so they add to 1:

\violet{p_i} = \frac{e^{\blue{z_i}}}{\sum_{j} e^{\blue{z_j}}}

Fig 5.8Drag the scores. Softmax keeps their order but turns them into a proper probability distribution.