Lesson 7.4MathCode~1 h

Backprop for whole layers

The three matrix formulas that train every dense layer — derived, checked numerically, and written by you.

Module 7 · 0/11 done

From single numbers to whole layers

Backpropagation one number at a time works, but it’s slow: a real network has millions of weights. The fix is to do the same calculus on whole matrices at once. A layer applied to a batch of examples is one matrix multiply:

Y = X\,W + \mathbf{b}

X: B \times n_{\text{in}} \quad W: n_{\text{in}} \times n_{\text{out}} \quad Y: B \times n_{\text{out}}

Going backwards, we receive $\partial L / \partial Y$ , how much the loss cares about each output, and need the gradients for $W$ , $\mathbf{b}$ and $X$ . The chain rule gives three short formulas, and there’s a trick to remembering them: every gradient has exactly the same shape as the thing it’s for. Only one arrangement of the pieces makes the shapes fit.

Fig 7.5Forward: X times W. Backward: each gradient is built from the incoming ∂L/∂Y and one transposed matrix — and comes out the same shape as its variable. Change the sizes and watch the shapes follow.