Convex Optimisation

Convex Optimisation #

Convex objectives have a single global minimum, making optimisation reliable.

Home | Continuous Optimisation

Nonlinear Optimisation

Nonlinear Optimisation in Machine Learning #

Practical training challenges and modern optimisers used in ML.

• Challenges in Gradient-Based Optimisation
• Stochastic Gradient Descent (SGD)
• Momentum-Based Learning
• Adaptive Methods: AdaGrad, RMSProp, Adam
• Tuning Hyperparameters and Preprocessing

Home | Calculus

Challenges in Gradient-Based Optimisation

Challenges in Gradient-Based Optimisation #

• Local optima and flat regions
• Differential curvature
• Difficult topologies (cliffs and valleys)

Home | Nonlinear Optimisation

Stochastic Gradient Descent (SGD)

Stochastic Gradient Descent (SGD) #

SGD uses mini-batches to trade exact gradients for speed and generalisation.

Home | Nonlinear Optimisation

Momentum-Based Learning

Momentum-Based Learning #

Momentum smooths updates and helps traverse valleys efficiently.

Home | Nonlinear Optimisation

Adaptive Methods: AdaGrad, RMSProp, Adam

Adaptive Methods: AdaGrad, RMSProp, Adam #

Adaptive methods adjust learning rates per-parameter.

Home | Nonlinear Optimisation

Tuning Hyperparameters and Preprocessing

Tuning Hyperparameters and Preprocessing #

• Learning rate schedules
• Initialisation
• Tuning hyperparameters
• Importance of feature preprocessing

Home | Nonlinear Optimisation

Dimensionality reduction and PCA

May 28, 2026

Dimensionality reduction and PCA #

PCA and SVM connect linear algebra, geometry, and optimisation.

Dimensionality reduction means representing high-dimensional data using fewer dimensions while trying to preserve the important structure of the data.

Principal Components Analysis, or PCA, is a linear dimensionality reduction method. It finds directions in the data along which the variance is maximum, and projects the data onto those directions.

Key takeaway: PCA chooses the eigenvectors of the covariance matrix corresponding to the largest eigenvalues. These eigenvectors form the principal subspace. The largest eigenvalues represent the directions that preserve the most variance.

AI Learning Resources

January 3, 2026

AI Learning Resources #

A curated list of high-quality online courses to learn Artificial Intelligence, Machine Learning, and Deep Learning from reputable universities and organisations.

Recommended Books & References #

Deep Neural Networks (DNN) #

• Deep Learning. MIT Press.
  Goodfellow, I., Bengio, Y., & Courville, A. (2016). (Vol. 1, No. 2).

• Introduction to Deep Learning. MIT Press.
  Eugene, C. (2019).

• Deep Learning with Python. Simon & Schuster.
  Chollet, F. (2021).

DNN Formula and Numerical Sheet

DNN Formula and Numerical Sheet #

This page consolidates the most useful Deep Neural Networks formulas and numerical patterns for revision.

It is designed for preparation and should be used together with the topic pages.

Revision strategy:
Do not only memorise formulas.

For each formula, know:

1. what each symbol means
2. when to apply it
3. how to substitute values carefully
4. what the output shape or answer represents

1. Artificial Neuron #

Weighted Sum ☆ #

Vector form: