Regularisation for Deep models
Regularisation for Deep models #
Regularisation means adding constraints or techniques that prevent a model from becoming too complex and memorising the training data.
The goal is not only low training error.
The goal is good performance on unseen data.
Key takeaway:
Regularisation helps the model generalise by controlling complexity, stabilising training, and reducing overfitting.
- Generalization for regression
- Training Error and Generalization Error
- Underfitting or Overfitting
- Model Selection
- Weight Decay and Norms
- Generalization in Classification
- Environment and Distribution Shift
- Generalization in Deep Learning
- Dropout
- Batch Normalization
- Layer Normalization
Underfitting, Good Fit, and Overfitting ☆ #
| Case | Model behaviour | Training error | Test error |
|---|---|---|---|
| Underfitting | too simple | high | high |
| Good fit | captures useful pattern | low | low |
| Overfitting | memorises training noise | very low | high |
flowchart LR
A["Model Complexity"] --> B["Too Simple: Underfitting"]
A --> C["Just Right: Good Fit"]
A --> D["Too Complex: Overfitting"]
style A fill:#E1F5FE,stroke:#4A90E2
style B fill:#FFF9C4,stroke:#FBC02D
style C fill:#C8E6C9,stroke:#43A047
style D fill:#EDE7F6,stroke:#7E57C2Training Error and Generalisation Error ☆ #
Training error measures performance on data used for learning.