Vector Calculus on Arshad Siddiqui

Differentiation of Univariate Functions

Mon, 01 Jan 0001 00:00:00 +0000

Differentiation of Univariate Functions #

Differentiation measures rate of change.

For a function f(x), the derivative measures the rate of change.

\[ f'(x) = \lim_{h \to 0} \frac{f(x+h)-f(x)}{h} \]

Interpretation:

Slope of tangent
Instantaneous rate of change

Home | Vector Calculus

Partial Differentiation and Gradients

Mon, 01 Jan 0001 00:00:00 +0000

Partial Differentiation and Gradients #

For f(x1, x2, …, xn):

[ \frac{\partial f}{\partial x_i} ]

Gradient vector:

[ \nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1} \ \vdots \ \frac{\partial f}{\partial x_n} \end{bmatrix} ]

Gradient points in direction of steepest ascent.

flowchart LR
 Input --> Function
 Function --> Gradient
 Gradient --> Optimisation

Home | Vector Calculus

Gradients of Vector-Valued and Matrix Functions

Mon, 01 Jan 0001 00:00:00 +0000

Gradients of Vector-Valued and Matrix Functions #

Covers gradients when outputs or parameters are vectors/matrices.

If f: R^n -> R^m, the derivative is the Jacobian.

[ J = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \dots & \frac{\partial f_1}{\partial x_n} \ \vdots & \ddots & \vdots \ \frac{\partial f_m}{\partial x_1} & \dots & \frac{\partial f_m}{\partial x_n} \end{bmatrix} ]

For scalar f(x):

[ H = \nabla^2 f ]

Hessian captures curvature.

Useful Gradient Identities

Mon, 01 Jan 0001 00:00:00 +0000

Useful Gradient Identities #

[ \nabla (a^T x) = a ] [ \nabla (x^T A x) = (A + A^T)x ]

If A symmetric:

[ \nabla (x^T A x) = 2Ax ]

These are heavily used in optimisation.

Home | Vector Calculus

Backpropagation and Automatic Differentiation

Mon, 01 Jan 0001 00:00:00 +0000

Backpropagation and Automatic Differentiation #

Backpropagation applies the chain rule:

efficiently across a computational graph.
repeatedly.

Chain rule:

[ \frac{dL}{dx} = \frac{dL}{dy} \cdot \frac{dy}{dx} ]

flowchart LR
 x --> y
 y --> L

Automatic differentiation computes exact derivatives efficiently using computational graphs.

Home | Vector Calculus

Higher-order derivatives

Mon, 01 Jan 0001 00:00:00 +0000

Higher-order derivatives #

Home | Vector Calculus

Taylor’s series

Mon, 01 Jan 0001 00:00:00 +0000

Linearization and multivariate Taylor’s series #

Home | Vector Calculus

Maxima and Minima

Mon, 01 Jan 0001 00:00:00 +0000

Computing maxima and minima for unconstrained optimization #

Home | Vector Calculus