Valleys, saddles & plateaus

Flat isn’t always the bottom

Gradient descent stops when the slope is zero. You might hope that means “lowest point”, but a flat spot can be three different things: the bottom of a valley (a minimum), the top of a hill (a maximum), or a ledge where the ground levels off and then keeps going down.

To tell them apart, look at how the slope is changing: the second derivative, written $f''(x)$. It measures the bend. Positive means the curve bends upwards like a cup; negative means it bends down like a cap.