Constrained Optimisation

Constrained Optimisation #

Optimisation with constraints (equalities/inequalities).

Home | Continuous Optimisation

Lagrange Multipliers

Lagrange Multipliers #

Transforms constrained problems into unconstrained ones using Lagrangians.

Home | Continuous Optimisation

Convex Optimisation

Convex Optimisation #

Convex objectives have a single global minimum, making optimisation reliable.

Home | Continuous Optimisation

Nonlinear Optimisation

Nonlinear Optimisation in Machine Learning #

Practical training challenges and modern optimisers used in ML.

• Challenges in Gradient-Based Optimisation
• Stochastic Gradient Descent (SGD)
• Momentum-Based Learning
• Adaptive Methods: AdaGrad, RMSProp, Adam
• Tuning Hyperparameters and Preprocessing

Home | Calculus

Challenges in Gradient-Based Optimisation

Challenges in Gradient-Based Optimisation #

• Local optima and flat regions
• Differential curvature
• Difficult topologies (cliffs and valleys)

Home | Nonlinear Optimisation

Stochastic Gradient Descent (SGD)

Stochastic Gradient Descent (SGD) #

SGD uses mini-batches to trade exact gradients for speed and generalisation.

Home | Nonlinear Optimisation

Momentum-Based Learning

Momentum-Based Learning #

Momentum smooths updates and helps traverse valleys efficiently.

Home | Nonlinear Optimisation

Adaptive Methods: AdaGrad, RMSProp, Adam

Adaptive Methods: AdaGrad, RMSProp, Adam #

Adaptive methods adjust learning rates per-parameter.

Home | Nonlinear Optimisation

Tuning Hyperparameters and Preprocessing

Tuning Hyperparameters and Preprocessing #

• Learning rate schedules
• Initialisation
• Tuning hyperparameters
• Importance of feature preprocessing

Home | Nonlinear Optimisation

Dimensionality reduction and PCA

May 28, 2026

Dimensionality reduction and PCA #

PCA and SVM connect linear algebra, geometry, and optimisation.

Dimensionality reduction means representing high-dimensional data using fewer dimensions while trying to preserve the important structure of the data.

Principal Components Analysis, or PCA, is a linear dimensionality reduction method. It finds directions in the data along which the variance is maximum, and projects the data onto those directions.

Key takeaway: PCA chooses the eigenvectors of the covariance matrix corresponding to the largest eigenvalues. These eigenvectors form the principal subspace. The largest eigenvalues represent the directions that preserve the most variance.