Lesson 7.8ModelCode~50 min
Regularisation
Weight decay, dropout, early stopping and data augmentation: four ways to stop a network memorising.
Module 7 · 0/11 done
Taming big models
Neural networks have so many parameters they can memorise their training set: noise, mistakes and all. You saw overfitting in Module 6. Regularisation is the toolbox for preventing it without making the model smaller.
- Weight decay: penalise big weights, so the model prefers smoother explanations.
- Early stopping: stop training when validation loss stops improving.
- Dropout: randomly switch off neurons while training.
- Data augmentation: make more training data by changing it slightly (shift, rotate, crop).