Special Matrices #
Certain types of matrices have special structural properties that are widely used in linear algebra and machine learning.
Identity Matrix #
An identity matrix is a square matrix where:
- all diagonal entries are 1
- all other entries are 0
Example (3×3):
\[ I = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \]Key Property #
\[ AI = IA = A \]Determinant #
\[ \det(I) = 1 \]Symmetric Matrix #
A matrix is symmetric if:
\[ A^T = A \]Used in
- Covariance matrices
- Quadratic forms
- Optimisation
The matrix is symmetric about its main diagonal.
Skew-Symmetric Matrix #
A matrix is skew-symmetric if:
\[ A^T = -A \]This implies:
\[ a_{ii} = 0 \]This implies all diagonal elements are zero.
Diagonal Matrix #
A matrix with non-zero elements only on its main diagonal.
\[ A = \begin{bmatrix} a_1 & 0 & 0 \\ 0 & a_2 & 0 \\ 0 & 0 & a_3 \end{bmatrix} \]Triangular Matrices #
Upper Triangular Matrix #
All elements below the main diagonal are zero:
\[ a_{ij} = 0 \quad \text{for } i > j \]Lower Triangular Matrix #
All elements above the main diagonal are zero:
\[ a_{ij} = 0 \quad \text{for } i < j \]Determinant of Triangular Matrices ⭐ #
For both upper and lower triangular matrices:
\[ \det(A) = \prod_{i=1}^{n} a_{ii} \]- The determinant equals the product of diagonal entries.
This makes computing determinants extremely fast for triangular matrices.
Inverse of Triangular Matrices ⭐ #
A triangular matrix is invertible if and only if:
\[ a_{ii} \neq 0 \quad \forall i \]If invertible:
\[ A^{-1} \text{ is also triangular of the same type} \]- Upper triangular → inverse is upper triangular
- Lower triangular → inverse is lower triangular
Strictly Lower Triangular Matrix #
All elements on and above the diagonal are zero:
\[ a_{ij} = 0 \quad \text{for } i \le j \]Unit Lower Triangular Matrix #
Diagonal entries are 1:
\[ a_{ii} = 1 \]Appears in LU decomposition.
Solving Systems with Triangular Matrices #
For lower triangular:
\[ A\mathbf{x} = \mathbf{b} \]Solve using forward substitution.
For upper triangular:
Use back substitution.
Positive Definite Matrix ⭐ #
A symmetric matrix (A) is positive definite if:
\[ \mathbf{x}^T A \mathbf{x} > 0 \quad \forall \mathbf{x} \neq 0 \]Positive Semi-Definite #
\[ \mathbf{x}^T A \mathbf{x} \ge 0 \]Key Properties #
If (A) is positive definite:
- All eigenvalues are positive
- Determinant is positive
- Matrix is invertible
- Cholesky decomposition exists
Why Positive Definite Matters in ML #
- Covariance matrices are positive semi-definite
- Hessian matrix in optimisation
- Guarantees convexity of quadratic functions
- Appears in Gaussian distributions
Example quadratic form:
\[ f(\mathbf{x}) = \mathbf{x}^T A \mathbf{x} \]If (A) is positive definite → function has a unique global minimum.
Sparse Matrix #
A matrix with mostly zero entries.
Used in
- Recommender systems
- Graph representations
- NLP
- Large-scale ML
Reduces:
- Memory
- Computation time