Lesson 6.3ModelCode~55 min

Many features at once

Predict from many inputs with one dot product — and see why features must be put on the same scale before training.

Module 6 · 0/11 done

More than one clue

A house’s price doesn’t depend on its size alone. Bedrooms, age and distance to the city centre all matter. With several features, the line becomes a weighted sum: one weight per feature, plus a bias. You’ve seen this shape before. It’s a dot product.

\amber{\hat{y}} = \green{w_1}\blue{x_1} + \green{w_2}\blue{x_2} + \dots + \green{w_d}\blue{x_d} + \green{b} = \green{\mathbf{w}} \cdot \blue{\mathbf{x}} + \green{b}

Training works exactly as before. Each weight gets its own partial derivative, and they all step downhill together. Let’s try it on 200 houses. Note the scales: sizes in the hundreds, bedrooms between 1 and 6.

Try itTraining on raw features

const { X, y, features } = data.housing(200)
console.log(features)
console.table(X.slice(0, 5).map((x, i) => ({ size: x[0], bedrooms: x[1], age: x[2], km: x[3], price: y[i] })))

// Gradient descent on the RAW numbers: sizes are ~100s, bedroom counts ~1–6.
// The size weight's gradient is huge, the others tiny — one learning rate can't suit them all.
for (const lr of [1e-6, 1e-5, 1e-4]) {
  let w = [0, 0, 0, 0], b = 0
  for (let epoch = 0; epoch < 200; epoch++) {
    const gw = [0, 0, 0, 0]
    let gb = 0
    for (let i = 0; i < X.length; i++) {
      const err = vec.dot(w, X[i]) + b - y[i]
      for (let j = 0; j < 4; j++) gw[j] += (2 * err * X[i][j]) / X.length
      gb += (2 * err) / X.length
    }
    w = w.map((v, j) => v - lr * gw[j])
    b -= lr * gb
  }
  const mse = vec.mean(X.map((x, i) => (vec.dot(w, x) + b - y[i]) ** 2))
  console.log("lr", lr, "→ error", round(mse, 1))
}

Press Run (⌘/Ctrl + Enter) to see output here.

A small learning rate barely moves. A slightly bigger one blows up. There’s no good setting. Why?