Vector Calculus on Arshad Siddiqui

Differentiation of Univariate Functions

Mon, 01 Jan 0001 00:00:00 +0000

Differentiation of Univariate Functions #

Differentiation measures rate of change.

For a function f(x), the derivative measures the rate of change.

$[ f'(x) = $lim_{h $to 0} $frac{f(x+h)-f(x)}{h} $]

Interpretation:

Slope of tangent
Instantaneous rate of change

Home | Vector Calculus

Partial Differentiation and Gradients

Mon, 01 Jan 0001 00:00:00 +0000

Partial Differentiation and Gradients #

For f(x1, x2, …, xn):

[ \frac{\partial f}{\partial x_i} ]

Gradient vector:

[ \nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1} \ \vdots \ \frac{\partial f}{\partial x_n} \end{bmatrix} ]

Gradient points in direction of steepest ascent.

flowchart LR
 Input --> Function
 Function --> Gradient
 Gradient --> Optimisation

Home | Vector Calculus

Gradients of Vector-Valued and Matrix Functions

Mon, 01 Jan 0001 00:00:00 +0000

Gradients of Vector-Valued and Matrix Functions #

Covers gradients when outputs or parameters are vectors/matrices.

If f: R^n -> R^m, the derivative is the Jacobian.

[ J = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \dots & \frac{\partial f_1}{\partial x_n} \ \vdots & \ddots & \vdots \ \frac{\partial f_m}{\partial x_1} & \dots & \frac{\partial f_m}{\partial x_n} \end{bmatrix} ]

For scalar f(x):

[ H = \nabla^2 f ]

Hessian captures curvature.

Useful Gradient Identities

Mon, 01 Jan 0001 00:00:00 +0000

Useful Gradient Identities #

[ \nabla (a^T x) = a ] [ \nabla (x^T A x) = (A + A^T)x ]

If A symmetric:

[ \nabla (x^T A x) = 2Ax ]

These are heavily used in optimisation.

Home | Vector Calculus

Backpropagation and Automatic Differentiation

Mon, 01 Jan 0001 00:00:00 +0000

Backpropagation and Automatic Differentiation #

Backpropagation applies the chain rule:

efficiently across a computational graph.
repeatedly.

Chain rule:

[ \frac{dL}{dx} = \frac{dL}{dy} \cdot \frac{dy}{dx} ]

flowchart LR
 x --> y
 y --> L

Automatic differentiation computes exact derivatives efficiently using computational graphs.

Home | Vector Calculus

Higher-order derivatives

Mon, 01 Jan 0001 00:00:00 +0000

Higher-order derivatives #

Home | Vector Calculus

Taylor’s series

Mon, 01 Jan 0001 00:00:00 +0000

Linearization and multivariate Taylor’s series #

Home | Vector Calculus

Maxima and Minima

Mon, 01 Jan 0001 00:00:00 +0000

Computing maxima and minima for unconstrained optimization #

Home | Vector Calculus

Vector Calculus

Mon, 01 Jan 0001 00:00:00 +0000

Vector Calculus #

Vector calculus extends differentiation to multivariate and vector-valued functions.

Gradients power learning. This section builds differentiation skills needed for backpropagation.

flowchart TD

 %% Core Node
 PD["Partial Derivatives"]

 %% Supporting Concepts
 DQ["Difference Quotient"]
 JH["Jacobian / Hessian"]
 TS["Taylor Series"]

 %% Application Chapters
 CH6["<br/>Probability"]
 CH7["<br/>Optimization"]
 CH9["<br/>Regression"]
 CH10["<br/>Dimensionality Reduction"]
 CH11["<br/>Density Estimation"]
 CH12["<br/>Classification"]

 %% Relationships
 DQ -->|defines| PD
 PD -->|collected in| JH
 JH -->|used in| TS
 JH -->|used in| CH6
	
 PD -->|used in| CH7
 PD -->|used in| CH9
 PD -->|used in| CH10
 PD -->|used in| CH11
 PD -->|used in| CH12

 %% Styling (Your Soft Academic Palette)
 style PD fill:#90CAF9,stroke:#1E88E5,color:#000

 style DQ fill:#CE93D8,stroke:#8E24AA,color:#000
 style JH fill:#CE93D8,stroke:#8E24AA,color:#000
 style TS fill:#CE93D8,stroke:#8E24AA,color:#000
 style CH6 fill:#CE93D8,stroke:#8E24AA,color:#000
	
 style CH7 fill:#C8E6C9,stroke:#2E7D32,color:#000
 style CH9 fill:#C8E6C9,stroke:#2E7D32,color:#000
 style CH10 fill:#C8E6C9,stroke:#2E7D32,color:#000
 style CH11 fill:#C8E6C9,stroke:#2E7D32,color:#000
 style CH12 fill:#C8E6C9,stroke:#2E7D32,color:#000

Home | Calculus