Differentiation measures rate of change.
For a function f(x), the derivative measures the rate of change.
$[ f'(x) = $lim_{h $to 0} $frac{f(x+h)-f(x)}{h} $]Interpretation:
For f(x1, x2, …, xn):
[ \frac{\partial f}{\partial x_i} ]Gradient vector:
[ \nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1} \ \vdots \ \frac{\partial f}{\partial x_n} \end{bmatrix} ]Gradient points in direction of steepest ascent.
flowchart LR
Input --> Function
Function --> Gradient
Gradient --> Optimisation
Covers gradients when outputs or parameters are vectors/matrices.
If f: R^n -> R^m, the derivative is the Jacobian.
[ J = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \dots & \frac{\partial f_1}{\partial x_n} \ \vdots & \ddots & \vdots \ \frac{\partial f_m}{\partial x_1} & \dots & \frac{\partial f_m}{\partial x_n} \end{bmatrix} ]For scalar f(x):
[ H = \nabla^2 f ]Hessian captures curvature.
If A symmetric:
[ \nabla (x^T A x) = 2Ax ]These are heavily used in optimisation.
Backpropagation applies the chain rule:
Chain rule:
[ \frac{dL}{dx} = \frac{dL}{dy} \cdot \frac{dy}{dx} ]
flowchart LR
x --> y
y --> L
Automatic differentiation computes exact derivatives efficiently using computational graphs.
Vector calculus extends differentiation to multivariate and vector-valued functions.
Gradients power learning. This section builds differentiation skills needed for backpropagation.
flowchart TD
%% Core Node
PD["Partial Derivatives"]
%% Supporting Concepts
DQ["Difference Quotient"]
JH["Jacobian / Hessian"]
TS["Taylor Series"]
%% Application Chapters
CH6["<br/>Probability"]
CH7["<br/>Optimization"]
CH9["<br/>Regression"]
CH10["<br/>Dimensionality Reduction"]
CH11["<br/>Density Estimation"]
CH12["<br/>Classification"]
%% Relationships
DQ -->|defines| PD
PD -->|collected in| JH
JH -->|used in| TS
JH -->|used in| CH6
PD -->|used in| CH7
PD -->|used in| CH9
PD -->|used in| CH10
PD -->|used in| CH11
PD -->|used in| CH12
%% Styling (Your Soft Academic Palette)
style PD fill:#90CAF9,stroke:#1E88E5,color:#000
style DQ fill:#CE93D8,stroke:#8E24AA,color:#000
style JH fill:#CE93D8,stroke:#8E24AA,color:#000
style TS fill:#CE93D8,stroke:#8E24AA,color:#000
style CH6 fill:#CE93D8,stroke:#8E24AA,color:#000
style CH7 fill:#C8E6C9,stroke:#2E7D32,color:#000
style CH9 fill:#C8E6C9,stroke:#2E7D32,color:#000
style CH10 fill:#C8E6C9,stroke:#2E7D32,color:#000
style CH11 fill:#C8E6C9,stroke:#2E7D32,color:#000
style CH12 fill:#C8E6C9,stroke:#2E7D32,color:#000