Constrained Optimisation
Constrained Optimisation #
Optimisation with constraints (equalities/inequalities).
Linear Algebra
Lagrange Multipliers
Lagrange Multipliers #
Transforms constrained problems into unconstrained ones using Lagrangians.
Convex Optimisation
Convex Optimisation #
Convex objectives have a single global minimum, making optimisation reliable.
Nonlinear Optimisation
Nonlinear Optimisation in Machine Learning #
Practical training challenges and modern optimisers used in ML.
- Challenges in Gradient-Based Optimisation
- Stochastic Gradient Descent (SGD)
- Momentum-Based Learning
- Adaptive Methods: AdaGrad, RMSProp, Adam
- Tuning Hyperparameters and Preprocessing
Challenges in Gradient-Based Optimisation
Challenges in Gradient-Based Optimisation #
- Local optima and flat regions
- Differential curvature
- Difficult topologies (cliffs and valleys)
Stochastic Gradient Descent (SGD)
Stochastic Gradient Descent (SGD) #
SGD uses mini-batches to trade exact gradients for speed and generalisation.
Momentum-Based Learning
Momentum-Based Learning #
Momentum smooths updates and helps traverse valleys efficiently.
Adaptive Methods: AdaGrad, RMSProp, Adam
Adaptive Methods: AdaGrad, RMSProp, Adam #
Adaptive methods adjust learning rates per-parameter.
Tuning Hyperparameters and Preprocessing
Tuning Hyperparameters and Preprocessing #
- Learning rate schedules
- Initialisation
- Tuning hyperparameters
- Importance of feature preprocessing
Dimensionality reduction and PCA
Dimensionality reduction and PCA #
PCA and SVM connect linear algebra, geometry, and optimisation.
- Principal Component Analysis (PCA)
- PCA Theory
- PCA in Practice
- Latent Variable Perspective
- Mathematical Preliminaries of SVM
- Nonlinear SVM and Kernels