Regularisation for Deep models

Regularisation for Deep models #

Regularisation means adding constraints or techniques that prevent a model from becoming too complex and memorising the training data.

The goal is not only low training error.

The goal is good performance on unseen data.

Key takeaway:
Regularisation helps the model generalise by controlling complexity, stabilising training, and reducing overfitting.

• Generalization for regression
• Training Error and Generalization Error
• Underfitting or Overfitting
• Model Selection
• Weight Decay and Norms
• Generalization in Classification
• Environment and Distribution Shift
• Generalization in Deep Learning
• Dropout
• Batch Normalization
• Layer Normalization

Underfitting, Good Fit, and Overfitting ☆ #

flowchart LR
    A["Model Complexity"] --> B["Too Simple: Underfitting"]
    A --> C["Just Right: Good Fit"]
    A --> D["Too Complex: Overfitting"]

    style A fill:#E1F5FE,stroke:#4A90E2
    style B fill:#FFF9C4,stroke:#FBC02D
    style C fill:#C8E6C9,stroke:#43A047
    style D fill:#EDE7F6,stroke:#7E57C2

Training Error and Generalisation Error ☆ #

Training error measures performance on data used for learning.

Linear Algebra

May 28, 2026

Linear Algebra #

The study of vectors and matrices is called Linear Algebra.

Linear Algebra provides the mathematical language used to represent data, transformations, and structure in ML.

• Linear Systems
  • Systems of Linear Equations
  • Matrices
  • Matrix Transposition
  • Solving Linear Systems
  • Forward and Backward Substitution
  • Inverse Matrix
  • Convex Combination
• Vector Spaces
  • Basis and Rank
  • Linear Independence
  • Norm
  • Inner Products and Dot Product
  • Lengths and Distances
  • Angles and Orthogonality
  • Orthonormal Basis
  • Feature Space
  • Cauchy–Schwarz
• Matrix Decompositions
  • Special Matrices
  • Characteristic Polynomial
  • Determinant and Trace
  • Eigenvalues and Eigenvectors
  • Cholesky Decomposition
  • Eigen Decomposition
  • Diagonalization
  • Singular Value Decomposition (SVD)
  • Matrix Approximation
• Dimensionality reduction and PCA
  • Primal and Dual Perspective for Linear SVM
  • Mathematical Preliminaries for SVM
  • Nonlinear SVM

Why Linear Algebra Matters in ML #

• Every machine learning model uses matrices
• All data in ML is represented using vectors and matrices
• Neural networks are pipelines of matrix operations
• Models apply matrix transformations to data
• Optimisation relies on linear algebra operations

What to Learn #

• Scalars, vectors, and matrices
• Vector operations (addition, dot product)
• Matrix multiplication (critical)
• Identity matrices and transpose
• Eigenvalues and eigenvectors (conceptual understanding)

• Scalar → a number
• Vector → a directed point
• Matrix → a space transformer
• Linear transformation → structured mapping
• Feature → one axis
• Feature space → where data lives
• Vector space → where vectors live

Home | Mathematical Foundation

Linear Systems

January 29, 2026

Linear Systems #

How systems of linear equations are represented and solved using matrices.

• the study of vectors and rules to manipulate vectors
• describe multiple linear equations solved simultaneously
• connect algebraic equations with matrix representations

Idea of Closure #

• performing a specific operation (like addition or multiplication) on members of a set always produces a result that belongs to the same set

• idea of closure is fundamental to defining a Vector space because it ensures that performing arithmetic operations (addition and scalar multiplication) on vectors within a set does not produce a new element outside that set.

Systems of Linear Equations

Systems of Linear Equations #

A system of linear equations can be written compactly as:

\[ A\mathbf{x}=\mathbf{b} \]

This represents:

• a linear transformation applied to an unknown vector (\mathbf{x})
• producing an output vector (\mathbf{b})

Key components #

Coefficient matrix (A) #

(A) contains the coefficients of the variables.

Calculus

Calculus #

Calculus is:

• the mathematical framework for understanding and controlling how quantities change
• the mathematics of change and accumulation

It helps answer:

• How fast is something changing right now?
• What happens when inputs change slightly?
• Where is something maximum or minimum?

It answers two big questions:

• How fast is something changing right now? → derivatives (differentiation)
• How much has accumulated over an interval? → integrals (integration)

flowchart TD
  A[Calculus] --> B[Limits]
  B --> C[Continuity]
  B --> D[Derivatives]
  B --> E[Integrals]
  D --> F[Optimisation: maxima/minima]
  D --> G[ML: gradients & learning]
  E --> H[Accumulation: area/total change]

• Vector Calculus
  • Differentiation of Univariate Functions
  • Partial Differentiation and Gradients
  • Gradients of Vector-Valued and Matrix Functions
  • Useful Gradient Identities
  • Backpropagation and Automatic Differentiation
  • Higher-order derivatives
  • Taylor’s series
  • Maxima and Minima
• Continuous Optimisation
  • Optimisation using Gradient Descent
  • Constrained Optimisation
  • Lagrange Multipliers
  • Convex Optimisation
• Nonlinear Optimisation
  • Challenges in Gradient-Based Optimisation
  • Stochastic Gradient Descent (SGD)
  • Momentum-Based Learning
  • Adaptive Methods: AdaGrad, RMSProp, Adam
  • Tuning Hyperparameters and Preprocessing

Matrices

Matrices #

Matrices are the core data structure of linear algebra and the workhorse of machine learning.
Almost every ML model can be described as a sequence of matrix operations.

• Special Matrices

Matrix #

A matrix is a rectangular array of numbers arranged in rows and columns.

\[ A \in \mathbb{R}^{m \times n} \]

An ( m \times n ) matrix has:

Solving Linear Systems

Solving Linear Systems #

Solve using:

• Substitution Method
• Elimination Method (Multiple & then Subtract)
• Cross Multiplication

Linear system can have:

• no solution
• a unique solution
• infinitely many solutions

Positive Definite Matrices #

A square matrix is positive definite if pre-multiplying and post-multiplying it by the same vector always gives a positive number as a result, independently of how we choose the vector.

Positive definite symmetric matrices have the property that all their eigenvalues are positive.

Forward and Backward Substitution

Forward and Backward Substitution #

Forward and backward substitution are efficient algorithms used to solve linear systems when the coefficient matrix is triangular.

They are typically used after:

• Gaussian elimination
• LU decomposition

1. Forward Substitution (Lower Triangular Systems) #

Used to solve:

\[ L\mathbf{x} = \mathbf{b} \]

where (L) is a lower triangular matrix:

Inverse Matrix

Inverse Matrix #

The inverse of a matrix is a matrix that, when multiplied with the original matrix, produces the identity matrix.

A square matrix (A) is invertible if there exists a matrix (A^{-1}) such that:

\[ AA^{-1} = A^{-1}A = I \]

Here:

Convex Combination

January 29, 2026

Convex Combination of Two Points #

A convex combination describes how to form a point between two points using weighted averages.

It is a fundamental building block in several advanced fields:

• Linear Algebra & Geometry
• Optimization Theory
• Machine Learning (Specifically in SVMs, clustering, and data interpolation)

Given two points (or vectors) $\mathbf{x}_1, \mathbf{x}_2 \in \mathbb{R}^n$, a convex combination of these points is defined as:

Where: