Linear Systems

January 29, 2026

Linear Systems #

How systems of linear equations are represented and solved using matrices.

• the study of vectors and rules to manipulate vectors
• describe multiple linear equations solved simultaneously
• connect algebraic equations with matrix representations

Idea of Closure #

• performing a specific operation (like addition or multiplication) on members of a set always produces a result that belongs to the same set

• idea of closure is fundamental to defining a Vector space because it ensures that performing arithmetic operations (addition and scalar multiplication) on vectors within a set does not produce a new element outside that set.

Systems of Linear Equations

Systems of Linear Equations #

A system of linear equations can be written compactly as:

\[ A\mathbf{x}=\mathbf{b} \]

This represents:

• a linear transformation applied to an unknown vector (\mathbf{x})
• producing an output vector (\mathbf{b})

Key components #

Coefficient matrix (A) #

(A) contains the coefficients of the variables.

Calculus

Calculus #

Calculus is:

• the mathematical framework for understanding and controlling how quantities change
• the mathematics of change and accumulation

It helps answer:

• How fast is something changing right now?
• What happens when inputs change slightly?
• Where is something maximum or minimum?

It answers two big questions:

• How fast is something changing right now? → derivatives (differentiation)
• How much has accumulated over an interval? → integrals (integration)

flowchart TD
  A[Calculus] --> B[Limits]
  B --> C[Continuity]
  B --> D[Derivatives]
  B --> E[Integrals]
  D --> F[Optimisation: maxima/minima]
  D --> G[ML: gradients & learning]
  E --> H[Accumulation: area/total change]

• Vector Calculus
  • Differentiation of Univariate Functions
  • Partial Differentiation and Gradients
  • Gradients of Vector-Valued and Matrix Functions
  • Useful Gradient Identities
  • Backpropagation and Automatic Differentiation
  • Higher-order derivatives
  • Taylor’s series
  • Maxima and Minima
• Continuous Optimisation
  • Optimisation using Gradient Descent
  • Constrained Optimisation
  • Lagrange Multipliers
  • Convex Optimisation
• Nonlinear Optimisation
  • Challenges in Gradient-Based Optimisation
  • Stochastic Gradient Descent (SGD)
  • Momentum-Based Learning
  • Adaptive Methods: AdaGrad, RMSProp, Adam
  • Tuning Hyperparameters and Preprocessing

Matrices

Matrices #

Matrices are the core data structure of linear algebra and the workhorse of machine learning.
Almost every ML model can be described as a sequence of matrix operations.

• Special Matrices

Matrix #

A matrix is a rectangular array of numbers arranged in rows and columns.

\[ A \in \mathbb{R}^{m \times n} \]

An ( m \times n ) matrix has:

Solving Linear Systems

Solving Linear Systems #

Solve using:

• Substitution Method
• Elimination Method (Multiple & then Subtract)
• Cross Multiplication

Linear system can have:

• no solution
• a unique solution
• infinitely many solutions

Positive Definite Matrices #

A square matrix is positive definite if pre-multiplying and post-multiplying it by the same vector always gives a positive number as a result, independently of how we choose the vector.

Positive definite symmetric matrices have the property that all their eigenvalues are positive.

Forward and Backward Substitution

Forward and Backward Substitution #

Forward and backward substitution are efficient algorithms used to solve linear systems when the coefficient matrix is triangular.

They are typically used after:

• Gaussian elimination
• LU decomposition

1. Forward Substitution (Lower Triangular Systems) #

Used to solve:

\[ L\mathbf{x} = \mathbf{b} \]

where (L) is a lower triangular matrix:

Inverse Matrix

Inverse Matrix #

The inverse of a matrix is a matrix that, when multiplied with the original matrix, produces the identity matrix.

A square matrix (A) is invertible if there exists a matrix (A^{-1}) such that:

\[ AA^{-1} = A^{-1}A = I \]

Here:

Convex Combination

January 29, 2026

Convex Combination of Two Points #

A convex combination describes how to form a point between two points using weighted averages.

It is a fundamental building block in several advanced fields:

• Linear Algebra & Geometry
• Optimization Theory
• Machine Learning (Specifically in SVMs, clustering, and data interpolation)

Given two points (or vectors) $\mathbf{x}_1, \mathbf{x}_2 \in \mathbb{R}^n$, a convex combination of these points is defined as:

Where:

Vector Spaces

Vector Spaces #

A vector space is the mathematical “home” where vectors live and where addition and scaling are valid operations.

• A vector space is a set closed under vector addition and scalar multiplication.

• Machine learning operates in vector spaces.

• covers independence, bases, rank, and geometric tools like norms and inner products that are used to measure length, distance, and angles.

A vector space is a set of vectors that follows ten axioms, defined under two operations:

Feature Space

Feature #

A feature is an individual measurable property or characteristic of a data point used as input to a machine learning model.

Each feature corresponds to one dimension.

A data point with ( d ) features is represented as: