Lesson 6.7ModelConcept~1 h 10 min

Overfitting & generalization

Memorising vs. understanding. Train, validation and test sets, and why simpler is often smarter.

Module 6 · 0/11 done

Memorising isn’t learning

Imagine two students. One memorises every answer in last year’s exam. The other understands the subject. On last year’s exam, both score 100%. On this year’s exam, only one of them does well.

Models face the same trap. What we actually care about is how well a model does on data it has never seen — that’s called generalisation. A model that fits its training data perfectly but fails on new data is overfitting.

Try itA model that memorises

// A "model" that just memorises: for a new x, return the y of the closest training point.
random.seed(1)
const all = data.line(60, 2, 1, 1.0)
const trainX = all.x.slice(0, 30), trainY = all.y.slice(0, 30)
const testX = all.x.slice(30), testY = all.y.slice(30)

const memorize = (x: number) => {
  let best = 0
  for (let i = 1; i < trainX.length; i++) if (Math.abs(trainX[i] - x) < Math.abs(trainX[best] - x)) best = i
  return trainY[best]
}
const line = (x: number) => 2 * x + 1

const mse = (f: (x: number) => number, xs: number[], ys: number[]) => vec.mean(xs.map((x, i) => (f(x) - ys[i]) ** 2))
console.log("memoriser — train error:", round(mse(memorize, trainX, trainY), 3), "  test error:", round(mse(memorize, testX, testY), 3))
console.log("simple line — train error:", round(mse(line, trainX, trainY), 3), "  test error:", round(mse(line, testX, testY), 3))

Press Run (⌘/Ctrl + Enter) to see output here.

The memoriser is perfect on training data (error 0!) and worse than a simple line on test data.