Gradient Descent

February 21, 2026

Gradient Descent for Linear Regression #

Revision:
Gradient descent is the step-by-step method for reducing the cost function when a direct closed-form solution is not convenient.

Where Gradient Descent Fits in ML ☆ #

Gradient descent is used when we want the model to learn parameters by repeatedly improving them.

For linear regression, it adjusts the slope and intercept until the prediction error becomes small.

flowchart LR
    A["Initial Parameters"] --> B["Make Predictions"]
    B --> C["Compute Cost"]
    C --> D["Compute Gradient"]
    D --> E["Update Parameters"]
    E --> B

    style A fill:#E1F5FE,stroke:#5b7db1,color:#000
    style B fill:#C8E6C9,stroke:#5f8f6a,color:#000
    style C fill:#FFF9C4,stroke:#b59b3b,color:#000
    style D fill:#EDE7F6,stroke:#8a6fb3,color:#000
    style E fill:#C8E6C9,stroke:#5f8f6a,color:#000

Core Idea ☆ #

The gradient tells us the direction in which the cost increases fastest.

Classification (Linear)

Linear models for Classification #

• categorises data by finding a linear boundary (hyperplane) that separates classes
• calculating a weighted sum of input features plus bias

flowchart TD
T["Linear<br/>classification<br/>models"] --> P["Perceptron"]
T --> LR["Logistic<br/>regression"]
T --> SVM["Linear<br/>SVM"]

P -->|uses| STEP["Step<br/>activation"]
LR -->|uses| SIG["Sigmoid<br/>+ log loss"]
SVM -->|uses| HNG["Hinge<br/>loss"]

style T fill:#90CAF9,stroke:#1E88E5,color:#000

style P fill:#C8E6C9,stroke:#2E7D32,color:#000
style LR fill:#C8E6C9,stroke:#2E7D32,color:#000
style SVM fill:#C8E6C9,stroke:#2E7D32,color:#000

style STEP fill:#CE93D8,stroke:#8E24AA,color:#000
style SIG fill:#CE93D8,stroke:#8E24AA,color:#000
style HNG fill:#CE93D8,stroke:#8E24AA,color:#000

• Discriminant Functions
• Decision Theory
• Probabilistic Discriminative Classifiers
• Logistic Regression

Logistic Regression #

• Supervised machine learning algorithm
• Binary classification algorithm
• requires data to be linearly separable
• predicts the probability that an input belongs to a specific class
• uses Sigmoid function to convert inputs into a probability value between 0 and 1

Key takeaway: Logistic regression predicts $P(y=1\mid x)$ using a sigmoid of a linear score $z=w\cdot x+b$, then learns $w,b$ by maximising likelihood (equivalently minimising log-loss).

Dynamic Programming

Dynamic Programming #

Dynamic Programming (DP) solves a reinforcement learning problem by repeatedly applying Bellman equations. It uses a complete model of the environment to calculate value functions and improve a policy.

• Dynamic Programming
• Policy Iteration
• Value Iteration
• Generalized policy iteration
• Efficiency of Dynamic Programming

1. When Dynamic Programming Can Be Used ☆ #

DP assumes that the environment dynamics are known. For every state and action, the agent needs the probabilities of possible next states and rewards:

Hypothesis Testing

Hypothesis Testing #

Hypothesis testing is a statistical decision-making method used to decide whether sample evidence is strong enough to reject an initial assumption about a population.

It connects probability, sampling distributions, confidence intervals, significance levels, and decision rules.

Key takeaway:
Hypothesis testing is not about proving something with certainty.

It is about asking:

If the null hypothesis were true, how surprising would this sample result be?

M4: Constrained Systems

M4: Constrained Systems #

This module considers how to make models practical when memory, compute, latency, and energy are limited. It connects model compression with edge deployment and the trade-offs required to retain useful prediction quality.

Module coverage:

• Model Optimisation Techniques: model compression, structured and unstructured pruning, post-training quantisation, and quantisation-aware training.
• Edge Deployment: TinyML, TensorFlow Lite, ONNX Runtime, and knowledge distillation.
• Performance Trade-offs: accuracy versus latency, model size versus throughput, and energy consumption optimisation.

Learning Objectives #

• Compare methods for reducing model size, computation, and memory use.
• Distinguish different quantisation and pruning approaches.
• Explain the role of knowledge distillation and deployment frameworks.
• Evaluate accuracy, latency, throughput, model size, and energy together.

Planned Topics #

This index records the module scope. Chapter pages can be added here as their source material becomes available.

Parallelisation of ML Algorithms

Parallelisation of ML Algorithms #

Machine-learning algorithms expose different kinds of parallel work. The correct decomposition depends on whether independent work occurs across records, features, trees, clusters, kernel entries, or optimisation steps.

Course coverage:

1. Problem decomposition for parallel machine learning
2. Ensemble methods and XGBoost-style tree ensembles
3. Parallel k-means assignment and centroid reduction
4. Distributed decision trees and random forests
5. Support vector machine parallelisation using block and MapReduce-style computation

Learning Objectives #

By the end of this page, you should be able to:

Foundation Models

December 14, 2025

Foundation Model #

AI models trained on massive datasets to perform a wide range of tasks with minimal fine-tuning.

• are large deep learning neural networks

• are large AI models trained on massive and diverse datasets (text, images, audio, or multiple modalities).

• Contain millions or billions of parameters.

• designed to perform a broad range of general tasks

• designed for general-purpose intelligence, not a single task.

• acts as base models for building specialised AI applications

LLM - Model

LLM – Large Language Model #

Large Language Models (LLMs) are advanced AI systems designed to process, understand, and generate human-like text.

They learn language by analysing massive amounts of text data, discovering patterns in:

• grammar

• meaning

• context

• relationships between words and sentences

• Built on Deep Learning

• Implemented using Neural Networks

• Based on Transformers

• Often combined with tools like:

  • Retrieval (RAG)
  • Agents
  • External APIs
  • Memory systems

What makes an LLM special? #

• Built using deep neural networks
• Trained on very large datasets (books, articles, code, web text)
• Can perform many tasks without task-specific training
• General-purpose language understanding, not single-task models

Foundation: Transformer Architecture #

LLMs are based on the Transformer Architecture, which allows models to understand context and long-range dependencies in text.

Communication-Aware Distributed ML

Communication-Aware Distributed ML #

Distributed machine learning is effective only when saved computation exceeds the cost of moving data and synchronising workers. Algorithm design must therefore account for message size, message frequency, barriers, and parameter placement.

Course coverage:

1. Communication overhead, illustrated through distributed k-means
2. Model parallelism when parameters do not fit or compute must be divided
3. Distributed k-nearest neighbours: partitioning, indexing, and approximate search
4. Gradient descent, SGD, and mini-batch optimisation
5. Synchronous versus asynchronous updates

Learning Objectives #

By the end of this page, you should be able to:

Decision Tree

Decision Tree #

A decision tree classifies an example by asking a sequence of questions about its attributes until it reaches a leaf (final decision).

Key takeaway: A decision tree grows by repeatedly splitting the training data into purer subsets using an impurity measure (Entropy / Gini / Classification Error).

• Information Theory
• Entropy Based Decision Tree Construction
• Avoiding Overfitting
• Minimum Description Length
• Handling Continuous valued attributes, missing attributes

Information Theory #

Decision trees need a way to measure: “How mixed are the class labels at a node?”