Momentum-Based Learning

Momentum-Based Learning #

Momentum smooths updates and helps traverse valleys efficiently.

Home | Nonlinear Optimisation

Adaptive Methods: AdaGrad, RMSProp, Adam

Adaptive Methods: AdaGrad, RMSProp, Adam #

Adaptive methods adjust learning rates per-parameter.

Home | Nonlinear Optimisation

Tuning Hyperparameters and Preprocessing

Tuning Hyperparameters and Preprocessing #

• Learning rate schedules
• Initialisation
• Tuning hyperparameters
• Importance of feature preprocessing

Home | Nonlinear Optimisation

Dimensionality reduction and PCA

May 28, 2026

Dimensionality reduction and PCA #

PCA and SVM connect linear algebra, geometry, and optimisation.

Dimensionality reduction means representing high-dimensional data using fewer dimensions while trying to preserve the important structure of the data.

Principal Components Analysis, or PCA, is a linear dimensionality reduction method. It finds directions in the data along which the variance is maximum, and projects the data onto those directions.

Key takeaway: PCA chooses the eigenvectors of the covariance matrix corresponding to the largest eigenvalues. These eigenvectors form the principal subspace. The largest eigenvalues represent the directions that preserve the most variance.