Linear Algebra on Arshad Siddiqui

Basis and Rank

Mon, 01 Jan 0001 00:00:00 +0000

Basis and Rank #

A basis is a minimal set of linearly independent vectors that spans a space.

The dimension of a space is the number of vectors in a basis.

Key Idea: Basis = independence + spanning. Rank tells us how many independent directions exist in a matrix.

A basis must satisfy two conditions ⭐

Vectors must be linearly independent
Vectors must span the space

This means:

No redundancy (independence)
Full coverage (spanning)

\[ \text{Span}(v_1, v_2, \dots, v_k) = V \] \[ c_1 v_1 + \cdots + c_k v_k = 0 \Rightarrow c_i = 0 \]

Why Basis Matters #

Represents space efficiently
Removes redundancy
Helps define coordinates
Used in ML for feature representation

Dimension #

Dimension is the number of vectors in a basis.

Linear Independence

Mon, 01 Jan 0001 00:00:00 +0000

Linear Independence #

A set of vectors is linearly independent if none of them can be written as a linear combination of the others.

\[ c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k = \mathbf{0} \;\Rightarrow\; c_1=\cdots=c_k=0 \]

Independence means each vector adds new information.

Why it matters #

Detects redundancy
Connects to rank and basis

If one vector can already be formed using others, it does not add anything new.

Norm

Mon, 01 Jan 0001 00:00:00 +0000

Norm #

A norm measures the length (magnitude) of a vector.

the norm of a vector x measures the distance from the origin to the point x.

Common example: Euclidean norm.

\[ \lVert \mathbf{x} \rVert_2 = \sqrt{x_1^2 + \cdots + x_n^2} \]

Key Idea: Norm = measure of size or length of a vector. It generalises the idea of distance in geometry to higher dimensions.

Common norms #

L1
L2
Infinity norm

Why it matters #

norms quantify size
are used in distances and regularisation.

Intuition (From Lectures) #

From lecture discussions on analytic geometry:

Inner Products and Dot Product

Mon, 01 Jan 0001 00:00:00 +0000

Inner Products and Dot Product #

An inner product maps two vectors to a single scalar.

It allows us to measure:

similarity
vector length
projections
orthogonality

flowchart TD
T["Inner<br/>products<br/>(types)"] --> DOT["Euclidean<br/>Dot product"]
T --> WIP["Weighted<br/>inner product"]
T --> FN["Function-space<br/>(integral)"]
T --> HERM["Complex<br/>Hermitian"]
T --> MAT["Matrix<br/>inner product<br/>(Frobenius)"]

DOT --> Rn["Vectors in<br/>
<span>
 \( \mathbb{R}^n \)
 </span>

"]
WIP --> SPD["SPD matrix<br/>W"]
FN --> L2["L2 space<br/>functions"]
HERM --> Cn["Vectors in<br/>C^n"]
MAT --> Mnm["Matrices<br/>R^{m×n}"]

style T fill:#90CAF9,stroke:#1E88E5,color:#000

style DOT fill:#C8E6C9,stroke:#2E7D32,color:#000
style WIP fill:#C8E6C9,stroke:#2E7D32,color:#000
style FN fill:#C8E6C9,stroke:#2E7D32,color:#000
style HERM fill:#C8E6C9,stroke:#2E7D32,color:#000
style MAT fill:#C8E6C9,stroke:#2E7D32,color:#000

style Rn fill:#CE93D8,stroke:#8E24AA,color:#000
style SPD fill:#CE93D8,stroke:#8E24AA,color:#000
style L2 fill:#CE93D8,stroke:#8E24AA,color:#000
style Cn fill:#CE93D8,stroke:#8E24AA,color:#000
style Mnm fill:#CE93D8,stroke:#8E24AA,color:#000

Definition #

For vectors
$ \mathbf{a}, \mathbf{b} \in \mathbb{R}^n $

Lengths and Distances

Mon, 01 Jan 0001 00:00:00 +0000

Lengths and Distances #

The length of a vector is given by its norm.

The distance between two points (vectors) is the norm of their difference.

Distance quantifies how far two vectors (data points) are from each other.

\[ d(\mathbf{x},\mathbf{y}) = \lVert \mathbf{x} - \mathbf{y} \rVert \]

Key Idea: Length measures size of a single vector. Distance measures separation between two vectors. Distance = norm applied to difference.

Angles and Orthogonality

Mon, 01 Jan 0001 00:00:00 +0000

Angles and Orthogonality #

Once we define an inner product, we can define the angle between two vectors.

Angles allow us to measure how aligned or different two vectors are in space.

Key Idea: Angle measures similarity between vectors. Orthogonality means complete independence (no similarity).

Why It Matters in Machine Learning #

PCA produces orthogonal components
Orthogonal features reduce redundancy
Gradient directions depend on angle

Angle Formula #

For vectors in n-dimensional space:

Orthonormal Basis

Mon, 01 Jan 0001 00:00:00 +0000

Orthonormal Basis #

A basis is orthonormal if its vectors are:

orthogonal to each other
each has unit length

\[ \langle \mathbf{e}_i, \mathbf{e}_j \rangle = \begin{cases} 1 & i=j \\ 0 & i\ne j \end{cases} \]

Key Idea: Orthonormal basis = perfectly independent + perfectly scaled. This makes computations extremely simple and stable.

Why it matters: orthonormal bases make projections and computations simple.

Intuition (From Lectures) #

From analytic geometry and vector space lectures:

Mathematical Foundation

Wed, 18 Mar 2026 00:00:00 +0000

Mathematical Foundations for Machine Learning #

Machine Learning is built on mathematical principles that allow models to:

represent data
learn patterns
optimise performance

flowchart LR
 DATA[Data]
 MATH[Math Models]
 OPT[Optimisation]
 MODEL[Trained Model]

 DATA --> MATH
 MATH --> OPT
 OPT --> MODEL

ML requires core mathematical tools to understand how ML algorithms work internally. Algebra deals with relationships between variables and quantities, while Calculus focuses on change and optimization.

Linear Algebra

Wed, 18 Mar 2026 00:00:00 +0000

Linear Algebra #

The study of vectors and matrices is called Linear Algebra.

Linear Algebra provides the mathematical language used to represent data, transformations, and structure in ML.

Why Linear Algebra Matters in ML #

Every machine learning model uses matrices
All data in ML is represented using vectors and matrices
Neural networks are pipelines of matrix operations
Models apply matrix transformations to data
Optimisation relies on linear algebra operations

What to Learn #

Scalars, vectors, and matrices
Vector operations (addition, dot product)
Matrix multiplication (critical)
Identity matrices and transpose
Eigenvalues and eigenvectors (conceptual understanding)

Scalar → a number
Vector → a directed point
Matrix → a space transformer
Linear transformation → structured mapping
Feature → one axis
Feature space → where data lives
Vector space → where vectors live

Home | Mathematical Foundation

Linear Systems

Thu, 29 Jan 2026 00:00:00 +0000

Linear Systems #

How systems of linear equations are represented and solved using matrices.

the study of vectors and rules to manipulate vectors
describe multiple linear equations solved simultaneously
connect algebraic equations with matrix representations

Idea of Closure #

performing a specific operation (like addition or multiplication) on members of a set always produces a result that belongs to the same set
idea of closure is fundamental to defining a Vector space because it ensures that performing arithmetic operations (addition and scalar multiplication) on vectors within a set does not produce a new element outside that set.

Systems of Linear Equations

Mon, 01 Jan 0001 00:00:00 +0000

Systems of Linear Equations #

A system of linear equations can be written compactly as:

\[ A\mathbf{x}=\mathbf{b} \]

This represents:

a linear transformation applied to an unknown vector (\mathbf{x})
producing an output vector (\mathbf{b})

Key components #

Coefficient matrix (A) #

(A) contains the coefficients of the variables.

Matrices

Mon, 01 Jan 0001 00:00:00 +0000

Matrices #

Matrices are the core data structure of linear algebra and the workhorse of machine learning.
Almost every ML model can be described as a sequence of matrix operations.

Special Matrices

Matrix #

A matrix is a rectangular array of numbers arranged in rows and columns.

\[ A \in \mathbb{R}^{m \times n} \]

An ( m \times n ) matrix has:

Solving Linear Systems

Mon, 01 Jan 0001 00:00:00 +0000

Solving Linear Systems #

Solve using:

Substitution Method
Elimination Method (Multiple & then Subtract)
Cross Multiplication

Linear system can have:

no solution
a unique solution
infinitely many solutions

Positive Definite Matrices #

A square matrix is positive definite if pre-multiplying and post-multiplying it by the same vector always gives a positive number as a result, independently of how we choose the vector.

Positive definite symmetric matrices have the property that all their eigenvalues are positive.

Forward and Backward Substitution

Mon, 01 Jan 0001 00:00:00 +0000

Forward and Backward Substitution #

Forward and backward substitution are efficient algorithms used to solve linear systems when the coefficient matrix is triangular.

They are typically used after:

Gaussian elimination
LU decomposition

1. Forward Substitution (Lower Triangular Systems) #

Used to solve:

\[ L\mathbf{x} = \mathbf{b} \]

where (L) is a lower triangular matrix:

Inverse Matrix

Mon, 01 Jan 0001 00:00:00 +0000

Inverse Matrix #

The inverse of a matrix is a matrix that, when multiplied with the original matrix, produces the identity matrix.

A square matrix (A) is invertible if there exists a matrix (A^{-1}) such that:

\[ AA^{-1} = A^{-1}A = I \]

Here:

Convex Combination

Thu, 29 Jan 2026 00:00:00 +0000

Convex Combination of Two Points #

A convex combination describes how to form a point between two points using weighted averages.

It is a fundamental building block in several advanced fields:

Linear Algebra & Geometry
Optimization Theory
Machine Learning (Specifically in SVMs, clustering, and data interpolation)

Given two points (or vectors) $\mathbf{x}_1, \mathbf{x}_2 \in \mathbb{R}^n$, a convex combination of these points is defined as:

$$\mathbf{x} = \lambda \mathbf{x}_1 + (1 - \lambda)\mathbf{x}_2$$

Where:

Vector Spaces

Mon, 01 Jan 0001 00:00:00 +0000

Vector Spaces #

A vector space is the mathematical “home” where vectors live and where addition and scaling are valid operations.

A vector space is a set closed under vector addition and scalar multiplication.
Machine learning operates in vector spaces.
covers independence, bases, rank, and geometric tools like norms and inner products that are used to measure length, distance, and angles.

A vector space is a set of vectors that follows ten axioms, defined under two operations:

Feature Space

Mon, 01 Jan 0001 00:00:00 +0000

Feature #

A feature is an individual measurable property or characteristic of a data point used as input to a machine learning model.

Each feature corresponds to one dimension.

\[ x_i \in \mathbb{R} \]

A data point with ( d ) features is represented as:

Cauchy–Schwarz

Mon, 01 Jan 0001 00:00:00 +0000

Cauchy–Schwarz Inequality #

The Cauchy–Schwarz Inequality is one of the most important results in linear algebra.

It places a fundamental bound on the inner product of two vectors.

If you see angle, cosine, similarity, or inner product bounds
→ think Cauchy–Schwarz Inequality

Key Idea: The inner product (dot product) can never exceed the product of magnitudes. This ensures all geometric interpretations (angles, cosine) are valid.

Statement of the Inequality #

For any vectors:

Matrix Decompositions

Wed, 18 Mar 2026 00:00:00 +0000

Matrix Decompositions #

Decompositions reveal structure in matrices and power algorithms like PCA.

Matrix decompositions break complex matrices into simpler parts.

From the lecture introduction, matrices are used to describe mappings and transformations of vectors.

That is why decomposition is important: it lets us understand a complicated transformation by rewriting it using simpler building blocks.

In the slides, the topic is introduced as part of three closely connected goals: how to summarise matrices, how matrices can be decomposed, and how the decompositions can be used for matrix approximations.

Characteristic Polynomial

Mon, 01 Jan 0001 00:00:00 +0000

Characteristic Polynomial #

The characteristic polynomial of a square matrix is the key tool used to compute eigenvalues.

It connects:

Determinants
Trace
Eigenvalues
Matrix structure

Definition #

Let
$ A \in \mathbb{R}^{n \times n} $
and $ \lambda \in \mathbb{R} $ .

The characteristic polynomial of (A) is defined as:

\[ p_A(\lambda) = \det(A - \lambda I) \]

It is a polynomial in $ \lambda $ of degree (n).

Determinant and Trace

Mon, 01 Jan 0001 00:00:00 +0000

Determinant and Trace #

Minor #

The minor of an element $ a_{ij} $ is the determinant of the smaller square matrix formed by:

removing row $ i $
removing column $ j $

The minor is denoted $ M_{ij} $ .

Minors are used to compute cofactors, which are used for determinants and inverses (via adjoint/adjugate).

Cofactor #

The cofactor of $ a_{ij} $ , denoted $ C_{ij} $ , is:

Eigenvalues and Eigenvectors

Mon, 01 Jan 0001 00:00:00 +0000

Eigenvalues and Eigenvectors #

Eigenvalues give scaling.
Eigenvectors define invariant directions of transformation.

Eigenvalues and eigenvectors describe directions that remain unchanged under a linear transformation, except for scaling.

From lectures: matrix multiplication represents a transformation of space.
Most vectors change direction and magnitude.
Some special vectors only scale.
These are eigenvectors.

Key Idea: A matrix transformation stretches or compresses vectors. Eigenvectors are directions that remain unchanged. Eigenvalues tell how much scaling happens.

Cholesky Decomposition

Mon, 01 Jan 0001 00:00:00 +0000

Cholesky Decomposition #

Cholesky decomposition is a special matrix factorisation used for symmetric positive definite matrices.

From lecture discussions, this decomposition is powerful because it reduces a matrix into a triangular form, making computations easier and more stable.

Key Idea: Cholesky decomposition expresses a matrix as a product of a lower triangular matrix and its transpose. It is efficient and numerically stable.

Definition #

For a symmetric positive definite matrix:

Eigen Decomposition

Mon, 01 Jan 0001 00:00:00 +0000

Eigen Decomposition #

Eigen decomposition expresses a matrix using its eigenvectors and eigenvalues.

From lecture discussions, this is one of the most important ways to understand the internal structure of a matrix.

Instead of treating the matrix as a black box, eigen decomposition reveals its fundamental directions and scaling behaviour.

Key Idea: Eigen decomposition rewrites a matrix in terms of directions (eigenvectors) and scaling factors (eigenvalues). This makes complex transformations easier to understand and compute.

Diagonalization

Mon, 01 Jan 0001 00:00:00 +0000

Diagonalization #

Diagonalisation expresses a matrix using its eigenvectors and eigenvalues when possible.

From lecture explanation, diagonalisation is one of the most powerful tools because it converts a complicated matrix into a much simpler form.

Instead of working with a full matrix, we work with a diagonal matrix, which is much easier to analyse and compute.

Key Idea: If a matrix has enough independent eigenvectors, it can be rewritten as a diagonal matrix using a change of basis. This simplifies matrix operations significantly.

Singular Value Decomposition (SVD)

Wed, 18 Mar 2026 00:00:00 +0000

Singular Value Decomposition (SVD) #

Singular Value Decomposition (SVD) is one of the most important matrix decomposition techniques in linear algebra and machine learning.

It factorises any matrix into three simpler matrices that reveal its structure.

Key Idea: SVD decomposes a matrix into rotations + scaling. It tells us how data is transformed along orthogonal directions.

Definition #

For any matrix in real space: \[ A \in \mathbb{R}^{m \times n} \]

Matrix Approximation

Mon, 01 Jan 0001 00:00:00 +0000

Matrix Approximation #

Low-rank approximation keeps the most important structure while reducing noise and computation.

Low-Rank Approximation #

Used for:

Dimensionality reduction
Noise removal
Efficient computation

Forms the basis of PCA.

Home | Matrix Decompositions

Continuous Optimisation

Mon, 01 Jan 0001 00:00:00 +0000

Continuous Optimisation #

Optimisation finds parameters that minimise (or maximise) an objective function.

Home | Calculus

Optimisation using Gradient Descent

Mon, 01 Jan 0001 00:00:00 +0000

Optimisation using Gradient Descent #

Gradient descent is an optimisation algorithm used to train ML and neural networks.

Gradient descent updates parameters by moving opposite the gradient.

Trains ML models by minimising errors:

between predicted and actual results
by iteratively adjusting its parameters
moves step‑by‑step in the direction of the steepest decrease in the loss function, it helps ML models learn the best possible weights for better predictions

Types of Gradient Gescent learning algorithms #

Batch gradient descent
Stochastic gradient descent
Mini-batch gradient descent

Home | Linear Algebra

Principal Component Analysis (PCA)

Mon, 01 Jan 0001 00:00:00 +0000

Principal Component Analysis (PCA) #

dimensionality reduction technique
helps us to reduce the number of features in a dataset while keeping the most important information.
changes complex datasets by transforming correlated features into a smaller set of uncorrelated components.
uses linear algebra to transform data into new features called principal components.
finds these by calculating eigenvectors (directions) and eigenvalues (importance) from the covariance matrix.
PCA selects the top components with the highest eigenvalues and projects the data onto them simplify the dataset.