Backpropagation

Every calculation is a graph

Write any calculation as a diagram: values flow left to right through little operations — multiply, add, square. This is a computational graph. Here’s $L = (a \cdot b + c)^2$, broken into steps: $d = a \cdot b$, then $e = d + c$, then $L = e^2$.

Computing the values left to right is the forward pass. Now the question that matters for learning: how much does L change if we nudge each of the inputs?