Build an autograd engine
Write the Value class that computes every gradient for you — like PyTorch, in 60 lines.
Numbers with a memory
Doing backprop by hand is fine for one formula. Real networks have millions of operations. So we automate it with a beautifully simple trick: wrap every number in an object that remembers which numbers it was made from and how to pass a gradient back to them.
Every time you add or multiply two of these objects, the result records its parents and a tiny “backward step” function. The calculation builds its own computational graph as it runs. Afterwards, one call to backward() walks that graph in reverse and fills in every gradient.
This is automatic differentiation (“autograd”), and it’s the engine inside PyTorch, JAX and TensorFlow. The version you’re about to write is the same idea in about 60 lines.