Mathematical Foundation

March 18, 2026

Mathematical Foundations for Machine Learning #

Machine Learning is built on mathematical principles that allow models to:

• represent data
• learn patterns
• optimise performance

flowchart LR
    DATA[Data]
    MATH[Math Models]
    OPT[Optimisation]
    MODEL[Trained Model]

    DATA --> MATH
    MATH --> OPT
    OPT --> MODEL

ML requires core mathematical tools to understand how ML algorithms work internally. Algebra deals with relationships between variables and quantities, while Calculus focuses on change and optimization.

Linear Algebra

March 18, 2026

Linear Algebra #

The study of vectors and matrices is called Linear Algebra.

Linear Algebra provides the mathematical language used to represent data, transformations, and structure in ML.

• Linear Systems
  • Systems of Linear Equations
  • Matrices
  • Matrix Transposition
  • Solving Linear Systems
  • Forward and Backward Substitution
  • Inverse Matrix
  • Convex Combination
• Vector Spaces
  • Basis and Rank
  • Linear Independence
  • Norm
  • Inner Products and Dot Product
  • Lengths and Distances
  • Angles and Orthogonality
  • Orthonormal Basis
  • Feature Space
  • Cauchy–Schwarz
• Matrix Decompositions
  • Special Matrices
  • Characteristic Polynomial
  • Determinant and Trace
  • Eigenvalues and Eigenvectors
  • Cholesky Decomposition
  • Eigen Decomposition
  • Diagonalization
  • Singular Value Decomposition (SVD)
  • Matrix Approximation
• Dimensionality reduction and PCA
  • Principal Component Analysis (PCA)
  • PCA Theory
  • PCA in Practice
  • Latent Variable Perspective
  • Mathematical Preliminaries of SVM
  • Nonlinear SVM and Kernels

Why Linear Algebra Matters in ML #

• Every machine learning model uses matrices
• All data in ML is represented using vectors and matrices
• Neural networks are pipelines of matrix operations
• Models apply matrix transformations to data
• Optimisation relies on linear algebra operations

What to Learn #

• Scalars, vectors, and matrices
• Vector operations (addition, dot product)
• Matrix multiplication (critical)
• Identity matrices and transpose
• Eigenvalues and eigenvectors (conceptual understanding)

• Scalar → a number
• Vector → a directed point
• Matrix → a space transformer
• Linear transformation → structured mapping
• Feature → one axis
• Feature space → where data lives
• Vector space → where vectors live

Home | Mathematical Foundation

Linear Systems

January 29, 2026

Linear Systems #

How systems of linear equations are represented and solved using matrices.

• the study of vectors and rules to manipulate vectors
• describe multiple linear equations solved simultaneously
• connect algebraic equations with matrix representations

Idea of Closure #

• performing a specific operation (like addition or multiplication) on members of a set always produces a result that belongs to the same set

• idea of closure is fundamental to defining a Vector space because it ensures that performing arithmetic operations (addition and scalar multiplication) on vectors within a set does not produce a new element outside that set.

Systems of Linear Equations

Systems of Linear Equations #

A system of linear equations can be written compactly as:

\[ A\mathbf{x}=\mathbf{b} \]

This represents:

• a linear transformation applied to an unknown vector (\mathbf{x})
• producing an output vector (\mathbf{b})

Key components #

Coefficient matrix (A) #

(A) contains the coefficients of the variables.

Calculus

Calculus #

Calculus is:

• the mathematical framework for understanding and controlling how quantities change
• the mathematics of change and accumulation

It helps answer:

• How fast is something changing right now?
• What happens when inputs change slightly?
• Where is something maximum or minimum?

It answers two big questions:

• How fast is something changing right now? → derivatives (differentiation)
• How much has accumulated over an interval? → integrals (integration)

flowchart TD
  A[Calculus] --> B[Limits]
  B --> C[Continuity]
  B --> D[Derivatives]
  B --> E[Integrals]
  D --> F[Optimisation: maxima/minima]
  D --> G[ML: gradients & learning]
  E --> H[Accumulation: area/total change]

• Vector Calculus
  • Differentiation of Univariate Functions
  • Partial Differentiation and Gradients
  • Gradients of Vector-Valued and Matrix Functions
  • Useful Gradient Identities
  • Backpropagation and Automatic Differentiation
  • Higher-order derivatives
  • Taylor’s series
  • Maxima and Minima
• Continuous Optimisation
  • Optimisation using Gradient Descent
  • Constrained Optimisation
  • Lagrange Multipliers
  • Convex Optimisation
• Nonlinear Optimisation
  • Challenges in Gradient-Based Optimisation
  • Stochastic Gradient Descent (SGD)
  • Momentum-Based Learning
  • Adaptive Methods: AdaGrad, RMSProp, Adam
  • Tuning Hyperparameters and Preprocessing

Matrices

Matrices #

Matrices are the core data structure of linear algebra and the workhorse of machine learning.
Almost every ML model can be described as a sequence of matrix operations.

• Special Matrices

Matrix #

A matrix is a rectangular array of numbers arranged in rows and columns.

\[ A \in \mathbb{R}^{m \times n} \]

An ( m \times n ) matrix has:

Solving Linear Systems

Solving Linear Systems #

Solve using:

• Substitution Method
• Elimination Method (Multiple & then Subtract)
• Cross Multiplication

Linear system can have:

• no solution
• a unique solution
• infinitely many solutions

Positive Definite Matrices #

A square matrix is positive definite if pre-multiplying and post-multiplying it by the same vector always gives a positive number as a result, independently of how we choose the vector.

Positive definite symmetric matrices have the property that all their eigenvalues are positive.

Forward and Backward Substitution

Forward and Backward Substitution #

Forward and backward substitution are efficient algorithms used to solve linear systems when the coefficient matrix is triangular.

They are typically used after:

• Gaussian elimination
• LU decomposition

1. Forward Substitution (Lower Triangular Systems) #

Used to solve:

\[ L\mathbf{x} = \mathbf{b} \]

where (L) is a lower triangular matrix:

Inverse Matrix

Inverse Matrix #

The inverse of a matrix is a matrix that, when multiplied with the original matrix, produces the identity matrix.

A square matrix (A) is invertible if there exists a matrix (A^{-1}) such that:

\[ AA^{-1} = A^{-1}A = I \]

Here:

Convex Combination

January 29, 2026

Convex Combination of Two Points #

A convex combination describes how to form a point between two points using weighted averages.

It is a fundamental building block in several advanced fields:

• Linear Algebra & Geometry
• Optimization Theory
• Machine Learning (Specifically in SVMs, clustering, and data interpolation)

Given two points (or vectors) $\mathbf{x}_1, \mathbf{x}_2 \in \mathbb{R}^n$, a convex combination of these points is defined as:

Where: