Lesson 7.7ModelMath~50 min
Making deep networks train
Vanishing and exploding signals, careful initialisation, normalisation and shortcuts — why depth used to fail.
Module 7 · 0/11 done
Autograd for whole matrices
Your Value engine tracks one number per node. For real networks, the course’s Tensor class is the same idea with a whole matrix in every node. Each operation (matmul, add, tanh…) records how to pass gradients back using the formulas from the layers lesson, and .backward() runs them in reverse order. It’s thousands of times faster.
// Your Value engine tracks single numbers. A Tensor is the same idea for whole matrices:
// every operation remembers how to pass gradients back, and .backward() runs the chain rule.
const X = Tensor.from([[1, 2], [3, -1], [0.5, 4]])
const W = Tensor.from([[0.3, -0.7, 0.2], [0.9, 0.1, -0.4]])
const b = Tensor.from([[0.1, 0.2, 0.3]])
const G = Tensor.from([[1, -2, 0.5], [0.3, 0.7, -1], [2, 0, 1]])
const Y = X.matmul(W).add(b) // the layer
const L = Y.mul(G).sum() // a made-up loss with ∂L/∂Y = G
L.backward()
// The Tensor engine agrees with the formula you just derived: dW = Xᵀ · G
console.log("W.grad: ", W.toArray().map((_, i) => Array.from(W.grad.slice(i * 3, i * 3 + 3))))
console.log("Xᵀ · G: ", mat.mul(mat.transpose(X.toArray()), G.toArray()))
console.log("b.grad: ", Array.from(b.grad), "(column sums of G)")
// Linear bundles W and b; Adam is a smarter optimiser (more in the optimizers lesson).
const layer = new Linear(196, 10)
console.log("a Linear(196, 10) has", layer.parameters().map(p => p.shape.join("×")).join(" and "), "parameters")
Press Run (⌘/Ctrl + Enter) to see output here.