Machine Learning on Arshad Siddiqui

Differentiation of Univariate Functions

Mon, 01 Jan 0001 00:00:00 +0000

Differentiation of Univariate Functions #

Differentiation measures rate of change.

For a function f(x), the derivative measures the rate of change.

$[ f'(x) = $lim_{h $to 0} $frac{f(x+h)-f(x)}{h} $]

Interpretation:

Slope of tangent
Instantaneous rate of change

Home | Vector Calculus

Partial Differentiation and Gradients

Mon, 01 Jan 0001 00:00:00 +0000

Partial Differentiation and Gradients #

For f(x1, x2, …, xn):

[ \frac{\partial f}{\partial x_i} ]

Gradient vector:

[ \nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1} \ \vdots \ \frac{\partial f}{\partial x_n} \end{bmatrix} ]

Gradient points in direction of steepest ascent.

flowchart LR
 Input --> Function
 Function --> Gradient
 Gradient --> Optimisation

Home | Vector Calculus

Linear Independence

Mon, 01 Jan 0001 00:00:00 +0000

Linear Independence #

A set of vectors is linearly independent if none of them can be written as a linear combination of the others.

\[ c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k = \mathbf{0} \;\Rightarrow\; c_1=\cdots=c_k=0 \]

Independence means each vector adds new information.

Why it matters #

Detects redundancy
Connects to rank and basis

If one vector can already be formed using others, it does not add anything new.

Gradients of Vector-Valued and Matrix Functions

Mon, 01 Jan 0001 00:00:00 +0000

Gradients of Vector-Valued and Matrix Functions #

Covers gradients when outputs or parameters are vectors/matrices.

If f: R^n -> R^m, the derivative is the Jacobian.

[ J = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \dots & \frac{\partial f_1}{\partial x_n} \ \vdots & \ddots & \vdots \ \frac{\partial f_m}{\partial x_1} & \dots & \frac{\partial f_m}{\partial x_n} \end{bmatrix} ]

For scalar f(x):

[ H = \nabla^2 f ]

Hessian captures curvature.

Reinforcement Learning

Mon, 01 Jan 0001 00:00:00 +0000

Reinforcement Learning (RL) #

RL is learning by trial and error.

Reinforcement Learning (RL) is a type of machine learning where an autonomous agent learns to make decisions by interacting with an environment.

Instead of being told the correct answer, the agent:

takes actions
observes outcomes
receives rewards or penalties
gradually learns a strategy that maximises long-term reward

Reinforcement Learning teaches an agent how to act, not what to predict.

Useful Gradient Identities

Mon, 01 Jan 0001 00:00:00 +0000

Useful Gradient Identities #

[ \nabla (a^T x) = a ] [ \nabla (x^T A x) = (A + A^T)x ]

If A symmetric:

[ \nabla (x^T A x) = 2Ax ]

These are heavily used in optimisation.

Home | Vector Calculus

Inner Products and Dot Product

Mon, 01 Jan 0001 00:00:00 +0000

Inner Products and Dot Product #

An inner product maps two vectors to a single scalar.

It allows us to measure:

similarity
vector length
projections
orthogonality

flowchart TD
T["Inner<br/>products<br/>(types)"] --> DOT["Euclidean<br/>Dot product"]
T --> WIP["Weighted<br/>inner product"]
T --> FN["Function-space<br/>(integral)"]
T --> HERM["Complex<br/>Hermitian"]
T --> MAT["Matrix<br/>inner product<br/>(Frobenius)"]

DOT --> Rn["Vectors in<br/>
<span>
 \( \mathbb{R}^n \)
 </span>

"]
WIP --> SPD["SPD matrix<br/>W"]
FN --> L2["L2 space<br/>functions"]
HERM --> Cn["Vectors in<br/>C^n"]
MAT --> Mnm["Matrices<br/>R^{m×n}"]

style T fill:#90CAF9,stroke:#1E88E5,color:#000

style DOT fill:#C8E6C9,stroke:#2E7D32,color:#000
style WIP fill:#C8E6C9,stroke:#2E7D32,color:#000
style FN fill:#C8E6C9,stroke:#2E7D32,color:#000
style HERM fill:#C8E6C9,stroke:#2E7D32,color:#000
style MAT fill:#C8E6C9,stroke:#2E7D32,color:#000

style Rn fill:#CE93D8,stroke:#8E24AA,color:#000
style SPD fill:#CE93D8,stroke:#8E24AA,color:#000
style L2 fill:#CE93D8,stroke:#8E24AA,color:#000
style Cn fill:#CE93D8,stroke:#8E24AA,color:#000
style Mnm fill:#CE93D8,stroke:#8E24AA,color:#000

Definition #

For vectors
$ \mathbf{a}, \mathbf{b} \in \mathbb{R}^n $

Backpropagation and Automatic Differentiation

Mon, 01 Jan 0001 00:00:00 +0000

Backpropagation and Automatic Differentiation #

Backpropagation applies the chain rule:

efficiently across a computational graph.
repeatedly.

Chain rule:

[ \frac{dL}{dx} = \frac{dL}{dy} \cdot \frac{dy}{dx} ]

flowchart LR
 x --> y
 y --> L

Automatic differentiation computes exact derivatives efficiently using computational graphs.

Home | Vector Calculus

Higher-order derivatives

Mon, 01 Jan 0001 00:00:00 +0000

Higher-order derivatives #

Home | Vector Calculus

Taylor’s series

Mon, 01 Jan 0001 00:00:00 +0000

Linearization and multivariate Taylor’s series #

Home | Vector Calculus

Maxima and Minima

Mon, 01 Jan 0001 00:00:00 +0000

Computing maxima and minima for unconstrained optimization #

Home | Vector Calculus

Naïve Bayes

Thu, 12 Mar 2026 00:00:00 +0000

Naïve Bayes #

Naïve Bayes is a probabilistic classifier.

Supervised Learning Problem
Binary Classification - final target variable is considered in two classes
Hypothesis is target which you want to classify
Total Probability (Prior) of Yes and No is already calculated
Post / Posterior is when you start studying data
Based on max probability of hypotheses classify given instance into a class

It predicts a class label by computing:

Mathematical Foundation

Wed, 18 Mar 2026 00:00:00 +0000

Mathematical Foundations for Machine Learning #

Machine Learning is built on mathematical principles that allow models to:

represent data
learn patterns
optimise performance

flowchart LR
 DATA[Data]
 MATH[Math Models]
 OPT[Optimisation]
 MODEL[Trained Model]

 DATA --> MATH
 MATH --> OPT
 OPT --> MODEL

ML requires core mathematical tools to understand how ML algorithms work internally. Algebra deals with relationships between variables and quantities, while Calculus focuses on change and optimization.

Linear Algebra

Wed, 18 Mar 2026 00:00:00 +0000

Linear Algebra #

The study of vectors and matrices is called Linear Algebra.

Linear Algebra provides the mathematical language used to represent data, transformations, and structure in ML.

Why Linear Algebra Matters in ML #

Every machine learning model uses matrices
All data in ML is represented using vectors and matrices
Neural networks are pipelines of matrix operations
Models apply matrix transformations to data
Optimisation relies on linear algebra operations

What to Learn #

Scalars, vectors, and matrices
Vector operations (addition, dot product)
Matrix multiplication (critical)
Identity matrices and transpose
Eigenvalues and eigenvectors (conceptual understanding)

Scalar → a number
Vector → a directed point
Matrix → a space transformer
Linear transformation → structured mapping
Feature → one axis
Feature space → where data lives
Vector space → where vectors live

Home | Mathematical Foundation

Linear Systems

Thu, 29 Jan 2026 00:00:00 +0000

Linear Systems #

How systems of linear equations are represented and solved using matrices.

the study of vectors and rules to manipulate vectors
describe multiple linear equations solved simultaneously
connect algebraic equations with matrix representations

Idea of Closure #

performing a specific operation (like addition or multiplication) on members of a set always produces a result that belongs to the same set
idea of closure is fundamental to defining a Vector space because it ensures that performing arithmetic operations (addition and scalar multiplication) on vectors within a set does not produce a new element outside that set.

Systems of Linear Equations

Mon, 01 Jan 0001 00:00:00 +0000

Systems of Linear Equations #

A system of linear equations can be written compactly as:

\[ A\mathbf{x}=\mathbf{b} \]

This represents:

a linear transformation applied to an unknown vector (\mathbf{x})
producing an output vector (\mathbf{b})

Key components #

Coefficient matrix (A) #

(A) contains the coefficients of the variables.

Matrices

Mon, 01 Jan 0001 00:00:00 +0000

Matrices #

Matrices are the core data structure of linear algebra and the workhorse of machine learning.
Almost every ML model can be described as a sequence of matrix operations.

Special Matrices

Matrix #

A matrix is a rectangular array of numbers arranged in rows and columns.

\[ A \in \mathbb{R}^{m \times n} \]

An ( m \times n ) matrix has:

Solving Linear Systems

Mon, 01 Jan 0001 00:00:00 +0000

Solving Linear Systems #

Solve using:

Substitution Method
Elimination Method (Multiple & then Subtract)
Cross Multiplication

Linear system can have:

no solution
a unique solution
infinitely many solutions

Positive Definite Matrices #

A square matrix is positive definite if pre-multiplying and post-multiplying it by the same vector always gives a positive number as a result, independently of how we choose the vector.

Positive definite symmetric matrices have the property that all their eigenvalues are positive.

Forward and Backward Substitution

Mon, 01 Jan 0001 00:00:00 +0000

Forward and Backward Substitution #

Forward and backward substitution are efficient algorithms used to solve linear systems when the coefficient matrix is triangular.

They are typically used after:

Gaussian elimination
LU decomposition

1. Forward Substitution (Lower Triangular Systems) #

Used to solve:

\[ L\mathbf{x} = \mathbf{b} \]

where (L) is a lower triangular matrix:

Inverse Matrix

Mon, 01 Jan 0001 00:00:00 +0000

Inverse Matrix #

The inverse of a matrix is a matrix that, when multiplied with the original matrix, produces the identity matrix.

A square matrix (A) is invertible if there exists a matrix (A^{-1}) such that:

\[ AA^{-1} = A^{-1}A = I \]

Here:

Vector Spaces

Mon, 01 Jan 0001 00:00:00 +0000

Vector Spaces #

A vector space is the mathematical “home” where vectors live and where addition and scaling are valid operations.

A vector space is a set closed under vector addition and scalar multiplication.
Machine learning operates in vector spaces.
covers independence, bases, rank, and geometric tools like norms and inner products that are used to measure length, distance, and angles.

A vector space is a set of vectors that follows ten axioms, defined under two operations:

Feature Space

Mon, 01 Jan 0001 00:00:00 +0000

Feature #

A feature is an individual measurable property or characteristic of a data point used as input to a machine learning model.

Each feature corresponds to one dimension.

\[ x_i \in \mathbb{R} \]

A data point with ( d ) features is represented as:

Cauchy–Schwarz

Mon, 01 Jan 0001 00:00:00 +0000

Cauchy–Schwarz Inequality #

The Cauchy–Schwarz Inequality is one of the most important results in linear algebra.

It places a fundamental bound on the inner product of two vectors.

If you see angle, cosine, similarity, or inner product bounds
→ think Cauchy–Schwarz Inequality

Key Idea: The inner product (dot product) can never exceed the product of magnitudes. This ensures all geometric interpretations (angles, cosine) are valid.

Statement of the Inequality #

For any vectors:

Matrix Decompositions

Wed, 18 Mar 2026 00:00:00 +0000

Matrix Decompositions #

Decompositions reveal structure in matrices and power algorithms like PCA.

Matrix decompositions break complex matrices into simpler parts.

From the lecture introduction, matrices are used to describe mappings and transformations of vectors.

That is why decomposition is important: it lets us understand a complicated transformation by rewriting it using simpler building blocks.

In the slides, the topic is introduced as part of three closely connected goals: how to summarise matrices, how matrices can be decomposed, and how the decompositions can be used for matrix approximations.

Characteristic Polynomial

Mon, 01 Jan 0001 00:00:00 +0000

Characteristic Polynomial #

The characteristic polynomial of a square matrix is the key tool used to compute eigenvalues.

It connects:

Determinants
Trace
Eigenvalues
Matrix structure

Definition #

Let
$ A \in \mathbb{R}^{n \times n} $
and $ \lambda \in \mathbb{R} $ .

The characteristic polynomial of (A) is defined as:

\[ p_A(\lambda) = \det(A - \lambda I) \]

It is a polynomial in $ \lambda $ of degree (n).

Determinant and Trace

Mon, 01 Jan 0001 00:00:00 +0000

Determinant and Trace #

Minor #

The minor of an element $ a_{ij} $ is the determinant of the smaller square matrix formed by:

removing row $ i $
removing column $ j $

The minor is denoted $ M_{ij} $ .

Minors are used to compute cofactors, which are used for determinants and inverses (via adjoint/adjugate).

Cofactor #

The cofactor of $ a_{ij} $ , denoted $ C_{ij} $ , is:

Eigenvalues and Eigenvectors

Mon, 01 Jan 0001 00:00:00 +0000

Eigenvalues and Eigenvectors #

Eigenvalues give scaling.
Eigenvectors define invariant directions of transformation.

Eigenvalues and eigenvectors describe directions that remain unchanged under a linear transformation, except for scaling.

From lectures: matrix multiplication represents a transformation of space.
Most vectors change direction and magnitude.
Some special vectors only scale.
These are eigenvectors.

Key Idea: A matrix transformation stretches or compresses vectors. Eigenvectors are directions that remain unchanged. Eigenvalues tell how much scaling happens.

Cholesky Decomposition

Mon, 01 Jan 0001 00:00:00 +0000

Cholesky Decomposition #

Cholesky decomposition is a special matrix factorisation used for symmetric positive definite matrices.

From lecture discussions, this decomposition is powerful because it reduces a matrix into a triangular form, making computations easier and more stable.

Key Idea: Cholesky decomposition expresses a matrix as a product of a lower triangular matrix and its transpose. It is efficient and numerically stable.

Definition #

For a symmetric positive definite matrix:

Eigen Decomposition

Mon, 01 Jan 0001 00:00:00 +0000

Eigen Decomposition #

Eigen decomposition expresses a matrix using its eigenvectors and eigenvalues.

From lecture discussions, this is one of the most important ways to understand the internal structure of a matrix.

Instead of treating the matrix as a black box, eigen decomposition reveals its fundamental directions and scaling behaviour.

Key Idea: Eigen decomposition rewrites a matrix in terms of directions (eigenvectors) and scaling factors (eigenvalues). This makes complex transformations easier to understand and compute.

Diagonalization

Mon, 01 Jan 0001 00:00:00 +0000

Diagonalization #

Diagonalisation expresses a matrix using its eigenvectors and eigenvalues when possible.

From lecture explanation, diagonalisation is one of the most powerful tools because it converts a complicated matrix into a much simpler form.

Instead of working with a full matrix, we work with a diagonal matrix, which is much easier to analyse and compute.

Key Idea: If a matrix has enough independent eigenvectors, it can be rewritten as a diagonal matrix using a change of basis. This simplifies matrix operations significantly.

Singular Value Decomposition (SVD)

Wed, 18 Mar 2026 00:00:00 +0000

Singular Value Decomposition (SVD) #

Singular Value Decomposition (SVD) is one of the most important matrix decomposition techniques in linear algebra and machine learning.

It factorises any matrix into three simpler matrices that reveal its structure.

Key Idea: SVD decomposes a matrix into rotations + scaling. It tells us how data is transformed along orthogonal directions.

Definition #

For any matrix in real space: \[ A \in \mathbb{R}^{m \times n} \]

Matrix Approximation

Mon, 01 Jan 0001 00:00:00 +0000

Matrix Approximation #

Low-rank approximation keeps the most important structure while reducing noise and computation.

Low-Rank Approximation #

Used for:

Dimensionality reduction
Noise removal
Efficient computation

Forms the basis of PCA.

Home | Matrix Decompositions

Vector Calculus

Mon, 01 Jan 0001 00:00:00 +0000

Vector Calculus #

Vector calculus extends differentiation to multivariate and vector-valued functions.

Gradients power learning. This section builds differentiation skills needed for backpropagation.

flowchart TD

 %% Core Node
 PD["Partial Derivatives"]

 %% Supporting Concepts
 DQ["Difference Quotient"]
 JH["Jacobian / Hessian"]
 TS["Taylor Series"]

 %% Application Chapters
 CH6["<br/>Probability"]
 CH7["<br/>Optimization"]
 CH9["<br/>Regression"]
 CH10["<br/>Dimensionality Reduction"]
 CH11["<br/>Density Estimation"]
 CH12["<br/>Classification"]

 %% Relationships
 DQ -->|defines| PD
 PD -->|collected in| JH
 JH -->|used in| TS
 JH -->|used in| CH6
	
 PD -->|used in| CH7
 PD -->|used in| CH9
 PD -->|used in| CH10
 PD -->|used in| CH11
 PD -->|used in| CH12

 %% Styling (Your Soft Academic Palette)
 style PD fill:#90CAF9,stroke:#1E88E5,color:#000

 style DQ fill:#CE93D8,stroke:#8E24AA,color:#000
 style JH fill:#CE93D8,stroke:#8E24AA,color:#000
 style TS fill:#CE93D8,stroke:#8E24AA,color:#000
 style CH6 fill:#CE93D8,stroke:#8E24AA,color:#000
	
 style CH7 fill:#C8E6C9,stroke:#2E7D32,color:#000
 style CH9 fill:#C8E6C9,stroke:#2E7D32,color:#000
 style CH10 fill:#C8E6C9,stroke:#2E7D32,color:#000
 style CH11 fill:#C8E6C9,stroke:#2E7D32,color:#000
 style CH12 fill:#C8E6C9,stroke:#2E7D32,color:#000

Home | Calculus

Continuous Optimisation

Mon, 01 Jan 0001 00:00:00 +0000

Continuous Optimisation #

Optimisation finds parameters that minimise (or maximise) an objective function.

Home | Calculus

Optimisation using Gradient Descent

Mon, 01 Jan 0001 00:00:00 +0000

Optimisation using Gradient Descent #

Gradient descent is an optimisation algorithm used to train ML and neural networks.

Gradient descent updates parameters by moving opposite the gradient.

Trains ML models by minimising errors:

between predicted and actual results
by iteratively adjusting its parameters
moves step‑by‑step in the direction of the steepest decrease in the loss function, it helps ML models learn the best possible weights for better predictions

Types of Gradient Gescent learning algorithms #

Batch gradient descent
Stochastic gradient descent
Mini-batch gradient descent

Home | Continuous Optimisation

Constrained Optimisation

Mon, 01 Jan 0001 00:00:00 +0000

Constrained Optimisation #

Optimisation with constraints (equalities/inequalities).

Home | Continuous Optimisation

Lagrange Multipliers

Mon, 01 Jan 0001 00:00:00 +0000

Lagrange Multipliers #

Transforms constrained problems into unconstrained ones using Lagrangians.

Home | Continuous Optimisation

Convex Optimisation

Mon, 01 Jan 0001 00:00:00 +0000

Convex Optimisation #

Convex objectives have a single global minimum, making optimisation reliable.

Home | Continuous Optimisation

Nonlinear Optimisation

Mon, 01 Jan 0001 00:00:00 +0000

Nonlinear Optimisation in Machine Learning #

Practical training challenges and modern optimisers used in ML.

Home | Calculus

Challenges in Gradient-Based Optimisation

Mon, 01 Jan 0001 00:00:00 +0000

Challenges in Gradient-Based Optimisation #

Local optima and flat regions
Differential curvature
Difficult topologies (cliffs and valleys)

Home | Nonlinear Optimisation

Stochastic Gradient Descent (SGD)

Mon, 01 Jan 0001 00:00:00 +0000

Stochastic Gradient Descent (SGD) #

SGD uses mini-batches to trade exact gradients for speed and generalisation.

Home | Nonlinear Optimisation

Momentum-Based Learning

Mon, 01 Jan 0001 00:00:00 +0000

Momentum-Based Learning #

Momentum smooths updates and helps traverse valleys efficiently.

Home | Nonlinear Optimisation

Adaptive Methods: AdaGrad, RMSProp, Adam

Mon, 01 Jan 0001 00:00:00 +0000

Adaptive Methods: AdaGrad, RMSProp, Adam #

Adaptive methods adjust learning rates per-parameter.

Home | Nonlinear Optimisation

Tuning Hyperparameters and Preprocessing

Mon, 01 Jan 0001 00:00:00 +0000

Tuning Hyperparameters and Preprocessing #

Learning rate schedules
Initialisation
Tuning hyperparameters
Importance of feature preprocessing

Home | Nonlinear Optimisation

Dimensionality reduction and PCA

Mon, 01 Jan 0001 00:00:00 +0000

Dimensionality reduction and PCA #

PCA and SVM connect linear algebra, geometry, and optimisation.

Home | Linear Algebra

Principal Component Analysis (PCA)

Mon, 01 Jan 0001 00:00:00 +0000

Principal Component Analysis (PCA) #

dimensionality reduction technique
helps us to reduce the number of features in a dataset while keeping the most important information.
changes complex datasets by transforming correlated features into a smaller set of uncorrelated components.
uses linear algebra to transform data into new features called principal components.
finds these by calculating eigenvectors (directions) and eigenvalues (importance) from the covariance matrix.
PCA selects the top components with the highest eigenvalues and projects the data onto them simplify the dataset.

PCA prioritizes the directions where the data varies the most because more variation = more useful information.

PCA Theory

Mon, 01 Jan 0001 00:00:00 +0000

PCA Theory #

Problem setting
Maximum variance perspective
Projection perspective
Eigenvector and low-rank approximations

Home | Dimensionality reduction and PCA

PCA in Practice

Mon, 01 Jan 0001 00:00:00 +0000

PCA in Practice #

Key steps of PCA in practice, including considerations in high dimensions.

Home | Dimensionality reduction and PCA

Latent Variable Perspective

Mon, 01 Jan 0001 00:00:00 +0000

Latent Variable Perspective #

PCA can be interpreted as modelling data using a smaller number of latent variables.

Home | Dimensionality reduction and PCA

Mathematical Preliminaries of SVM

Mon, 01 Jan 0001 00:00:00 +0000

Mathematical Preliminaries of SVM #

Primal and dual perspectives
Geometry of margins

Home | Dimensionality reduction and PCA

Nonlinear SVM and Kernels

Mon, 01 Jan 0001 00:00:00 +0000

Nonlinear SVM and Kernels #

Kernels allow inner products in high-dimensional feature spaces without explicit mapping.

Home | Dimensionality reduction and PCA

AI Learning Resources

Sat, 03 Jan 2026 12:00:00 +0100

AI Learning Resources #

A curated list of high-quality online courses to learn Artificial Intelligence, Machine Learning, and Deep Learning from reputable universities and organisations.

Recommended Books & References #

Deep Neural Networks (DNN) #

Deep Learning. MIT Press.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). (Vol. 1, No. 2).
Introduction to Deep Learning. MIT Press.
Eugene, C. (2019).
Deep Learning with Python. Simon & Schuster.
Chollet, F. (2021).

ML Pipeline

Tue, 21 Apr 2026 00:00:00 +0000

Machine Learning Pipeline: Preprocessing & Models #

This page explains both data preprocessing and model development concepts in a clear, structured way to support understanding.

A complete ML pipeline includes preprocessing, feature engineering, feature selection, and model training.

1. Data Preprocessing Overview #

Raw data is often:

Noisy
Incomplete
Inconsistent

Preprocessing ensures data is suitable for machine learning.

2. Missing Values #

Why they occur

Sensor errors
Data collection issues

Methods