Linear Systems on Arshad Siddiqui

Systems of Linear Equations

Mon, 01 Jan 0001 00:00:00 +0000

Systems of Linear Equations #

A system of linear equations can be written compactly as:

\[ A\mathbf{x}=\mathbf{b} \]

This represents:

a linear transformation applied to an unknown vector (\mathbf{x})
producing an output vector (\mathbf{b})

Key components #

Coefficient matrix (A) #

(A) contains the coefficients of the variables.

Matrices

Mon, 01 Jan 0001 00:00:00 +0000

Matrices #

Matrices are the core data structure of linear algebra and the workhorse of machine learning.
Almost every ML model can be described as a sequence of matrix operations.

Special Matrices

Matrix #

A matrix is a rectangular array of numbers arranged in rows and columns.

\[ A \in \mathbb{R}^{m \times n} \]

An ( m \times n ) matrix has:

Matrix Transposition

Mon, 01 Jan 0001 00:00:00 +0000

Transposition of a Matrix #

The transpose of a matrix is obtained by swapping rows and columns.

\[ A = [a_{ij}] \]

then the transpose of ( A ), denoted ( A^T ), is:

\[ A^T = [a_{ji}] \]

Rules of Matrix Transposition #

1. Transpose of a Transpose #

\[ (A^T)^T = A \]

Intuition
Swapping rows and columns twice returns the original matrix.

Solving Linear Systems

Mon, 01 Jan 0001 00:00:00 +0000

Solving Linear Systems #

Solve using:

Substitution Method
Elimination Method (Multiple & then Subtract)
Cross Multiplication

Linear system can have:

no solution
a unique solution
infinitely many solutions

Positive Definite Matrices #

A square matrix is positive definite if pre-multiplying and post-multiplying it by the same vector always gives a positive number as a result, independently of how we choose the vector.

Positive definite symmetric matrices have the property that all their eigenvalues are positive.

Forward and Backward Substitution

Mon, 01 Jan 0001 00:00:00 +0000

Forward and Backward Substitution #

Forward and backward substitution are efficient algorithms used to solve linear systems when the coefficient matrix is triangular.

They are typically used after:

Gaussian elimination
LU decomposition

1. Forward Substitution (Lower Triangular Systems) #

Used to solve:

\[ L\mathbf{x} = \mathbf{b} \]

where (L) is a lower triangular matrix:

Inverse Matrix

Mon, 01 Jan 0001 00:00:00 +0000

Inverse Matrix #

The inverse of a matrix is a matrix that, when multiplied with the original matrix, produces the identity matrix.

A square matrix (A) is invertible if there exists a matrix (A^{-1}) such that:

\[ AA^{-1} = A^{-1}A = I \]

Here:

Convex Combination

Thu, 29 Jan 2026 00:00:00 +0000

Convex Combination of Two Points #

A convex combination describes how to form a point between two points using weighted averages.

It is a fundamental building block in several advanced fields:

Linear Algebra & Geometry
Optimization Theory
Machine Learning (Specifically in SVMs, clustering, and data interpolation)

Given two points (or vectors) $\mathbf{x}_1, \mathbf{x}_2 \in \mathbb{R}^n$, a convex combination of these points is defined as:

$$\mathbf{x} = \lambda \mathbf{x}_1 + (1 - \lambda)\mathbf{x}_2$$

Where: