Optimizers: SGD to Adam

Plain gradient descent has problems

Real loss landscapes aren’t neat bowls. They have long narrow ravines — steep in one direction, nearly flat in another. Gradient descent bounces between the steep walls while barely creeping along the floor. One learning rate can’t suit both directions: big enough to make progress along the floor means too big for the walls.

A handful of smarter update rules — optimizers — fix this.