Gradient Descent

February 21, 2026

Gradient Descent for Linear Regression #

Revision:
Gradient descent is the step-by-step method for reducing the cost function when a direct closed-form solution is not convenient.

Where Gradient Descent Fits in ML ☆ #

Gradient descent is used when we want the model to learn parameters by repeatedly improving them.

For linear regression, it adjusts the slope and intercept until the prediction error becomes small.

flowchart LR
    A["Initial Parameters"] --> B["Make Predictions"]
    B --> C["Compute Cost"]
    C --> D["Compute Gradient"]
    D --> E["Update Parameters"]
    E --> B

    style A fill:#E1F5FE,stroke:#5b7db1,color:#000
    style B fill:#C8E6C9,stroke:#5f8f6a,color:#000
    style C fill:#FFF9C4,stroke:#b59b3b,color:#000
    style D fill:#EDE7F6,stroke:#8a6fb3,color:#000
    style E fill:#C8E6C9,stroke:#5f8f6a,color:#000

Core Idea ☆ #

The gradient tells us the direction in which the cost increases fastest.

LNN for Classification

February 15, 2026

Linear NN for Classification #

A Linear Neural Network (LNN) for classification uses no hidden layers.
It learns a linear decision boundary and outputs class probabilities, then converts them into predicted classes.

Neural-network view:

• Binary classification → logistic regression (single neuron + sigmoid)
• Multi-class classification → softmax regression (K output neurons + softmax)

flowchart LR
  D["Data<br/>X, y"] --> M["Linear model<br/>w, b"]
  M --> A["Activation<br/>Sigmoid / Softmax"]
  A --> L["Loss<br/>Cross-entropy"]
  L --> O["Optimiser<br/>Mini-batch GD / Adam"]
  O --> P["Updated parameters<br/>w, b"]
  P --> I["Inference<br/>Probabilities → class"]

  %% Pastel colour scheme
  style D fill:#E3F2FD,stroke:#1E88E5,stroke-width:1px
  style M fill:#E8F5E9,stroke:#43A047,stroke-width:1px
  style A fill:#FFF3E0,stroke:#FB8C00,stroke-width:1px
  style L fill:#FCE4EC,stroke:#D81B60,stroke-width:1px
  style O fill:#F3E5F5,stroke:#8E24AA,stroke-width:1px
  style P fill:#E0F7FA,stroke:#00838F,stroke-width:1px
  style I fill:#F1F8E9,stroke:#558B2F,stroke-width:1px

Classification #

Classification predicts a discrete class label.
Common settings:

Classification (Linear)

Linear models for Classification #

• categorises data by finding a linear boundary (hyperplane) that separates classes
• calculating a weighted sum of input features plus bias

flowchart TD
T["Linear<br/>classification<br/>models"] --> P["Perceptron"]
T --> LR["Logistic<br/>regression"]
T --> SVM["Linear<br/>SVM"]

P -->|uses| STEP["Step<br/>activation"]
LR -->|uses| SIG["Sigmoid<br/>+ log loss"]
SVM -->|uses| HNG["Hinge<br/>loss"]

style T fill:#90CAF9,stroke:#1E88E5,color:#000

style P fill:#C8E6C9,stroke:#2E7D32,color:#000
style LR fill:#C8E6C9,stroke:#2E7D32,color:#000
style SVM fill:#C8E6C9,stroke:#2E7D32,color:#000

style STEP fill:#CE93D8,stroke:#8E24AA,color:#000
style SIG fill:#CE93D8,stroke:#8E24AA,color:#000
style HNG fill:#CE93D8,stroke:#8E24AA,color:#000

• Discriminant Functions
• Decision Theory
• Probabilistic Discriminative Classifiers
• Logistic Regression

Logistic Regression #

• Supervised machine learning algorithm
• Binary classification algorithm
• requires data to be linearly separable
• predicts the probability that an input belongs to a specific class
• uses Sigmoid function to convert inputs into a probability value between 0 and 1

Key takeaway: Logistic regression predicts $P(y=1\mid x)$ using a sigmoid of a linear score $z=w\cdot x+b$, then learns $w,b$ by maximising likelihood (equivalently minimising log-loss).

Dynamic Programming

Dynamic Programming #

Dynamic Programming (DP) solves a reinforcement learning problem by repeatedly applying Bellman equations. It uses a complete model of the environment to calculate value functions and improve a policy.

• Dynamic Programming
• Policy Iteration
• Value Iteration
• Generalized policy iteration
• Efficiency of Dynamic Programming

1. When Dynamic Programming Can Be Used ☆ #

DP assumes that the environment dynamics are known. For every state and action, the agent needs the probabilities of possible next states and rewards:

Hypothesis Testing

Hypothesis Testing #

Hypothesis testing is a statistical decision-making method used to decide whether sample evidence is strong enough to reject an initial assumption about a population.

It connects probability, sampling distributions, confidence intervals, significance levels, and decision rules.

Key takeaway:
Hypothesis testing is not about proving something with certainty.

It is about asking:

If the null hypothesis were true, how surprising would this sample result be?

Informed Search and Heuristic Design

Informed Search and Heuristic Design #

Informed search uses additional knowledge to estimate which states are most promising. A good heuristic can greatly reduce unnecessary exploration while preserving solution quality.

• Notion of Heuristics
• Informed Search Algorithms : Greedy best first search, A* search
• Optimality of A*
• Heuristic Functions
• Heuristic Accuracy & Algorithm performance
• Admissible heuristics from relaxed problems, pattern databases. Experience.
• Use case : Learning Heuristics using Supervised Learning

Learning Objectives #

• explain the purpose of a heuristic function
• compare Greedy Best-First Search and A* Search
• trace A* using OPEN and CLOSED lists
• test heuristics for admissibility and consistency
• explain heuristic dominance and effective branching factor
• derive heuristics from relaxed problems and subproblems
• describe pattern databases, landmarks and learned heuristics

1. What Is Informed Search? ☆ #

Uninformed search knows only how to generate successors and recognise a goal. Informed search additionally estimates whether one non-goal state is more promising than another.

M4: Constrained Systems

M4: Constrained Systems #

This module considers how to make models practical when memory, compute, latency, and energy are limited. It connects model compression with edge deployment and the trade-offs required to retain useful prediction quality.

Module coverage:

• Model Optimisation Techniques: model compression, structured and unstructured pruning, post-training quantisation, and quantisation-aware training.
• Edge Deployment: TinyML, TensorFlow Lite, ONNX Runtime, and knowledge distillation.
• Performance Trade-offs: accuracy versus latency, model size versus throughput, and energy consumption optimisation.

Learning Objectives #

• Compare methods for reducing model size, computation, and memory use.
• Distinguish different quantisation and pruning approaches.
• Explain the role of knowledge distillation and deployment frameworks.
• Evaluate accuracy, latency, throughput, model size, and energy together.

Planned Topics #

This index records the module scope. Chapter pages can be added here as their source material becomes available.

NN and Neural Language Modelling

Neural Networks and Neural Language Modelling #

Neural networks learn useful representations and nonlinear relationships directly from data. In language modelling, they replace discrete N-gram identities with learned word embeddings and use these representations to predict the next word.

Learning Objectives #

• Explain the computation performed by a neural unit.
• Describe why hidden layers and nonlinear activations are needed.
• Explain how feed-forward networks support NLP classification.
• Trace the flow through a feed-forward neural language model.
• Compare N-gram and neural language models.

Big Picture #

flowchart TD
    A["Context Words"] --> B["One-hot Inputs"]
    B --> C["Embedding Lookup"]
    C --> D["Combined Context"]
    D --> E["Hidden Layer"]
    E --> F["Softmax"]
    F --> G["Next-word Probabilities"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9
    style G fill:#FFF9C4

1. Neural Network Units ☆ #

A neural unit receives input values, multiplies them by learned weights, adds a bias, and applies an activation function.

Parallelisation of ML Algorithms

Parallelisation of ML Algorithms #

Machine-learning algorithms expose different kinds of parallel work. The correct decomposition depends on whether independent work occurs across records, features, trees, clusters, kernel entries, or optimisation steps.

Course coverage:

1. Problem decomposition for parallel machine learning
2. Ensemble methods and XGBoost-style tree ensembles
3. Parallel k-means assignment and centroid reduction
4. Distributed decision trees and random forests
5. Support vector machine parallelisation using block and MapReduce-style computation

Learning Objectives #

By the end of this page, you should be able to:

Foundation Models

December 14, 2025

Foundation Model #

AI models trained on massive datasets to perform a wide range of tasks with minimal fine-tuning.

• are large deep learning neural networks

• are large AI models trained on massive and diverse datasets (text, images, audio, or multiple modalities).

• Contain millions or billions of parameters.

• designed to perform a broad range of general tasks

• designed for general-purpose intelligence, not a single task.

• acts as base models for building specialised AI applications