Lesson 6.6ModelCode~55 min

Measuring what matters

Why accuracy lies. Confusion matrices, precision, recall, thresholds and ROC curves.

Module 6 · 0/11 done

When accuracy lies

Accuracy, the fraction of answers that are right, sounds like the obvious score. But suppose 1 in 20 transactions is fraud and a model simply says “not fraud” every time:

Try itThe lazy fraud detector

// 1 in 20 transactions is fraud. A "model" that ALWAYS says "not fraud":
const truth = range(1000).map(i => (i % 20 === 0 ? 1 : 0))
const lazy = truth.map(() => 0)

const accuracy = truth.filter((t, i) => t === lazy[i]).length / truth.length
console.log("accuracy:", accuracy)                                       // 95%!
console.log("frauds caught:", truth.filter((t, i) => t === 1 && lazy[i] === 1).length, "of", truth.filter(t => t === 1).length)

Press Run (⌘/Ctrl + Enter) to see output here.

95% accuracy, and it catches nothing. When one class is rare, accuracy hides the failures that matter. We need sharper tools.