Lesson 6.5ModelCode~50 min

Many classes: softmax regression

From yes/no to “which of these?”: one score per class, softmax, and cross-entropy.

Module 6 · 0/11 done

More than yes or no

Logistic regression answers yes-or-no questions. But which digit is this, 0 to 9? Which of 50 000 words comes next? For $K$ classes we give the model $K$ scores (one weighted sum per class) and turn them into probabilities with softmax, from Module 5.

z_k = \mathbf{w}_k \cdot \mathbf{x} + b_k

\mathbf{p} = \text{softmax}(\mathbf{z})

All the weights together form a matrix $W$ : one row per input feature, one column per class. The loss is cross-entropy, and the gradient is just as tidy as before: predicted probabilities minus the true answer.

\frac{\partial L}{\partial z_k} = p_k - [k = y]

Fig 6.5Three classes, three scores per point. Press Train: each colour’s region is where its score wins. The borders between regions are always straight lines.