AI

Markov Decision Processes

AI, ML, Reinforcement Learning, Markov Decision Process, MDP, Bellman Equation

Markov Decision Processes #

A Markov Decision Process (MDP) is the mathematical framework used to describe sequential decision-making in reinforcement learning. It brings together the agent, environment, states, actions, transition dynamics, rewards, returns, policies, and value functions in one model.

A basic bandit asks which action is best in a recurring situation. An MDP goes further: the action taken now can change the next state, which changes the decisions and rewards that become possible later.

N-gram Language Modelling

AI, ML

Natural Language Processing, NLP, Language Modelling, N-Grams, Smoothing, Perplexity

N-gram Language Modelling #

A language model assigns probabilities to sequences of words. It can compare complete sentences or predict which word is likely to come next.

Key ideas include:

word prediction and sequence probability
the chain rule and Markov assumption
unigram, bigram and trigram models
Maximum Likelihood Estimation
unseen sequences and smoothing
interpolation and backoff
intrinsic and extrinsic evaluation
perplexity

Learning Objectives #

Explain what a language model represents.
Calculate simple unigram and bigram probabilities.
Explain why unseen N-grams create zero probabilities.
Distinguish smoothing, interpolation and backoff.
Interpret perplexity correctly.

Big Picture #

flowchart TD
    A["Training Corpus"] --> B["Count N-grams"]
    B --> C["Estimate Probabilities"]
    C --> D["Handle Unseen Events"]
    D --> E["Score Word Sequences"]
    E --> F["Evaluate Model"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9

1. What Is a Language Model? ☆ #

A language model estimates how probable a sequence of words is.

Parallel Programming Models

AI, ML

ML System Optimization, MLSysOps, Multi-Core CPU, GPGPU, SIMD, MIMD, SIMT, GPU, TPU, AI, ML

Parallel Programming Models #

Parallel algorithms need hardware that can execute independent work efficiently. Modern systems therefore combine multiple CPU cores, memory hierarchies, threads, instruction pipelines, GPUs, clusters, and specialised matrix processors.

Course coverage:

Parallel programming models for expressing independent work
MapReduce, task-parallel, and request-parallel patterns
Multi-core CPUs and GPGPUs: SIMD, MIMD, SIMT, memory hierarchy, and bandwidth
Distributed clusters: CPU-only and GPU-accelerated systems, data sharding, parameter servers, and all-reduce

The hardware sections also cover specialised matrix processors and the performance limits introduced by memory and communication.

Problem Solving and Uninformed Search

AI, Artificial and Computational Intelligence

AI, ACI, Problem Solving, Uninformed Search, BFS, DFS, UCS, IDS

Problem Solving and Uninformed Search #

A problem-solving agent converts a goal into a precisely defined search problem. An uninformed search algorithm then explores the state space using only the information in that problem definition.

Problem Solving Agent
Problem Formulation - Examples
From Uninformed to Informed Search Algorithms

Learning Objectives #

formulate a search problem using five components
distinguish a state-space graph from a search tree
explain nodes, frontiers and node expansion
compare tree search with graph search when states repeat
compare BFS, DFS, UCS, DLS and IDS
select an appropriate uninformed strategy for a problem
distinguish offline and online search

Big Picture #

flowchart TD
    G["Goal"] --> F["Formulate problem"]
    F --> S["Search for actions"]
    S --> X["Execute solution"]

    style G fill:#E1F5FE
    style F fill:#C8E6C9
    style S fill:#FFF9C4
    style X fill:#EDE7F6

1. Problem-Solving Agents ☆ #

A problem-solving agent normally follows three stages:

Regression (Linear)

AI, ML

Linear Regression #

Linear Regression is a supervised ML method used to predict a numerical target by fitting a model that is linear in its parameters.

In ML , linear models are a core baseline: they’re fast, often surprisingly strong, and usually easy to interpret.

Key takeaway: Linear Regression learns parameters by minimising a squared-error cost. You can solve it directly (closed form) or iteratively (gradient descent), and you can extend it using basis functions and regularisation.

Random Variables

February 22, 2026

AI, Statistics

AI, Statistics, Probability

Random Variables #

A random variable is a way to attach numbers to outcomes of a random experiment.

It lets us move from: “what happened?” to: “what number should we analyse?”

Key takeaway: A random variable is a function from the sample space to real numbers. Once you define the random variable clearly, the rest (pmf/pdf/cdf, mean, variance) becomes systematic.

flowchart TD
PD["Probability<br/>distributions"] --> RV["Random<br/>variables"]

RV --> T["Types"]
T --> RV1["Discrete<br/>RVs"]
T --> RV2["Continuous<br/>RVs"]

RV --> F["PMF / PDF / CDF"]
RV --> S["Mean / Variance<br/>Covariance"]
RV --> J["Joint & Marginal<br/>distributions"]
RV --> X["Transformations"]

style PD fill:#90CAF9,stroke:#1E88E5,color:#000
style RV fill:#90CAF9,stroke:#1E88E5,color:#000

style T fill:#CE93D8,stroke:#8E24AA,color:#000
style F fill:#CE93D8,stroke:#8E24AA,color:#000
style S fill:#CE93D8,stroke:#8E24AA,color:#000
style J fill:#CE93D8,stroke:#8E24AA,color:#000
style X fill:#CE93D8,stroke:#8E24AA,color:#000
style RV1 fill:#CE93D8,stroke:#8E24AA,color:#000
style RV2 fill:#CE93D8,stroke:#8E24AA,color:#000

1) Definition #

Random variable: a rule that assigns a number to each outcome.

Common Probability Distributions

February 22, 2026

AI, Statistics

AI, Statistics, Probability

Common Probability Distributions #

Once you can describe a random variable using a pmf or pdf, the next step is to use named distributions that appear repeatedly in real data and in ML models.

Key takeaway: Named distributions give you ready-made probability models for common patterns: binary outcomes, counts, and measurement noise.

flowchart TD
PD["Probability<br/>distributions"] --> DS["Common<br/>distributions"]

DS --> DIS["Discrete"]
DS --> CON["Continuous"]

DIS --> D1["Bernoulli"]
DIS --> D2["Binomial"]
DIS --> D3["Poisson"]

CON --> D4["Normal<br/>(Gaussian)"]
CON --> D5["t / Chi-square / F<br/>(intro)"]

style PD fill:#90CAF9,stroke:#1E88E5,color:#000
style DS fill:#90CAF9,stroke:#1E88E5,color:#000

style DIS fill:#CE93D8,stroke:#8E24AA,color:#000
style CON fill:#CE93D8,stroke:#8E24AA,color:#000

style D1 fill:#C8E6C9,stroke:#2E7D32,color:#000
style D2 fill:#C8E6C9,stroke:#2E7D32,color:#000
style D3 fill:#C8E6C9,stroke:#2E7D32,color:#000
style D4 fill:#C8E6C9,stroke:#2E7D32,color:#000
style D5 fill:#C8E6C9,stroke:#2E7D32,color:#000

1) Bernoulli distribution (binary) #

Use when: one trial has two outcomes (success/failure).

Ordinary Least Squares

February 21, 2026

AI, ML

AI, ML, Linear Regression, OLS

Direct solution method - Ordinary Least Squares and the Line of Best Fit #

Revision:
OLS is the direct method for linear regression. It finds the best-fit line by minimising the sum of squared residuals without iterative updates.

Direct Method vs Iterative Method ☆ #

Linear regression parameters can be found in two main ways.

Method	Main idea	When used
Ordinary Least Squares	Compute the best parameters directly	Small or moderate datasets
Gradient Descent	Start with parameters and update repeatedly	Large datasets or many features

flowchart LR
    A["Linear Regression"] --> B["Direct Solution<br/>OLS"]
    A --> C["Iterative Solution<br/>Gradient Descent"]

    B --> B1["Normal Equation"]
    B --> B2["No learning rate"]
    B --> B3["One-shot solution"]

    C --> C1["Learning rate"]
    C --> C2["Repeated updates"]
    C --> C3["Stops after convergence"]

    style A fill:#E1F5FE,stroke:#5b7db1,color:#000
    style B fill:#C8E6C9,stroke:#5f8f6a,color:#000
    style C fill:#FFF9C4,stroke:#b59b3b,color:#000
    style B1 fill:#EDE7F6,stroke:#8a6fb3,color:#000
    style B2 fill:#EDE7F6,stroke:#8a6fb3,color:#000
    style B3 fill:#EDE7F6,stroke:#8a6fb3,color:#000
    style C1 fill:#EDE7F6,stroke:#8a6fb3,color:#000
    style C2 fill:#EDE7F6,stroke:#8a6fb3,color:#000
    style C3 fill:#EDE7F6,stroke:#8a6fb3,color:#000

Why It Is Called “Least Squares” ☆ #

OLS is called least squares because it chooses parameters that make the squared residual errors as small as possible.

Cost Function

February 21, 2026

AI, ML

AI, ML, Linear Regression, OLS

Cost Function #

Revision:
A cost function converts model error into a single number. Training means changing the model parameters until this number becomes as small as possible.

Why Cost Function Matters in ML ☆ #

A machine learning model needs a way to decide whether one set of parameters is better than another.

For linear regression, every possible value of the parameters gives a different line. The cost function tells us which line is better by measuring how far the predictions are from the true values.

Gradient Descent Algorithm

February 26, 2026

AI, Machine Learning

Deep Learning, Optimisation

Gradient Descent Algorithm #

Gradient Descent Algorithm (GDA) is

an optimisation method
used to train models
by repeatedly updating parameters (weights and biases) to reduce the loss

In deep learning, the default training approach is almost always mini-batch gradient descent, usually with Adam or SGD + momentum.

Gradient Descent is used in both regression and classification.

It’s not tied to the task type — it’s tied to the fact you have: